|
Alec Matusis
matusis at yahoo.com
Tue Nov 6 21:42:35 EST 2007
> So, what is 'modules' and '_import' and how are they used?
'modules' is 'from sys import modules', so it's sys.modules...
'_import' is used in the publisher like so:
publisher.py:
from common._import import _import
def handler(req):
...
dyn_path = req.uri[1:] #to get rid of '/' prefix
module_path = string.replace(dyn_path,'/','.')
runme = _import( module_path )
....
common/_import.py:
from sys import modules
def _import( module_path ):
runme = modules[ module_path ]
#get the last part
class_name = module_path.split('.')[-1]
try:
runme = getattr(runme, class_name ) <--- INTERMITTENT ERROR here
except AttributeError:
raise Exception,'Module does not contain class '+class_name+',
runme='+repr(runme)+' module_path='+repr(module_path)
> How do you
> ensure that two concurrently executing request handlers trying to
> import the same module don't interfere with each other?
We do not insure that- the code was written for prefork MPM.
What is the best way to insure that?
> -----Original Message-----
> From: Graham Dumpleton [mailto:graham.dumpleton at gmail.com]
> Sent: Tuesday, November 06, 2007 3:53 AM
> To: Alec Matusis
> Cc: Martijn Moeling; mod_python at modpython.org
> Subject: Re: [mod_python] mod_python or apache scalability?
>
> On 06/11/2007, Alec Matusis <matusis at yahoo.com> wrote:
> > I found that this that
> >
> > IOError: image file is truncated (14432 bytes not processed)
> >
> > has to do with using input filter
> >
> > PythonInputFilter flashfilter FLASHFILTER
> > SetInputFilter FLASHFILTER
> >
> > It somehow behaves differently in apache 2.2.6/mod_python 3.3.1
> versus
> > 2.0.54/ 3.1.4. For now, I removed this input filter (we can get away
> without
> > it), and it seems to work.
> >
> > However, we have a bigger problem: we now have intermittent import
> errors
> > (one error per about 50-80 requests).
> >
> > This code executed by publisher:
> >
> > import modules
> > def _import( module_path ):
> > runme = modules[ module_path ]
> > #get the last part
> > class_name = module_path.split('.')[-1]
> > try:
> > runme = getattr(runme, class_name ) <--- INTERMITTENT ERROR
> here
> > except AttributeError:
> > raise Exception,'Module does not contain class
> '+class_name+',
> > runme='+repr(runme)+' module_path='+repr(module_path)
> >
> > produces this error:
> >
> > Exception: Module does not contain class getprofile, runme=<module
> > 'getprofile' from '/path/scripts/getprofile.py'>
> module_path='getprofile'
> >
> > This used to always work with prefork. Is there anything not thread
> safe
> > about importing modules?
>
> The underlying Python and mod_python import mechanisms themselves are
> never usually the problem. It is thread unsafe practices regarding
> initialisation in the code around the imports.
>
> So, if you have built your own form of module importer or higher level
> layer for one and you haven't properly thread protected it, it is most
> likely the cause.
>
> So, what is 'modules' and '_import' and how are they used? How do you
> ensure that two concurrently executing request handlers trying to
> import the same module don't interfere with each other?
>
> Graham
>
> > > -----Original Message-----
> > > From: mod_python-bounces at modpython.org [mailto:mod_python-
> > > bounces at modpython.org] On Behalf Of Graham Dumpleton
> > > Sent: Monday, November 05, 2007 1:29 PM
> > > To: Martijn Moeling
> > > Cc: mod_python at modpython.org; Alec Matusis
> > > Subject: Re: [mod_python] mod_python or apache scalability?
> > >
> > > The LimitRequestBody directive support in mod_python is broken and
> > > most likely would have resulted in a 500 error occurring during
> > > mod_python.publisher processing of the form even before the user
> code
> > > was run.
> > >
> > > My guess at the reason is that they have enabled in the new Apache
> > > server compression on request content. This may not work with
> > > mod_python because how it reads request content is broken, with it
> > > only reading up until original Content-Length in certain cases. If
> > > there is a mutating input filter such as content decompression, you
> > > will get truncated input data.
> > >
> > > https://issues.apache.org/jira/browse/MODPYTHON-240
> > > https://issues.apache.org/jira/browse/MODPYTHON-212
> > >
> > > So, OP should ensure that directives to accept compressed request
> > > content are disabled.
> > >
> > > Graham
> > >
> > > On 05/11/2007, Martijn Moeling <martijn at xs4us.nu> wrote:
> > > >
> > > >
> > > >
> > > > Hmm it looks to me that you have an upload limit in your apache
> > > config.
> > > >
> > > > It might be the LimitRequestBody directive.
> > > >
> > > > Another thing might be that your processing is faster than the
> data
> > > comes
> > > > in, but I doubt that
> > > > Take a close look at you apache conf, and your apache error log
> > > (right after
> > > > an upload or with tail -f /var/log/httpd/error_log (or where ever
> > > your
> > > > logdir is)).
> > > >
> > > > Martijn
> > > >
> > > >
> > > > ________________________________
> > > > Van: Alec Matusis [mailto:matusis at matusis.com]
> > > > Verzonden: zo 04.11.2007 21:46
> > > > Aan: Martijn Moeling
> > > > CC: mod_python at modpython.org
> > > > Onderwerp: RE: [mod_python] mod_python or apache scalability?
> > > >
> > > >
> > > >
> > > >
> > > > > I had similair problems however, they turned out to be MySQLdb
> > > > > related and not in a way you suspect
> > > >
> > > > Thanks! I will try separating MySQLdb connections. We do pass the
> > > connection
> > > > around currently.
> > > >
> > > > However, how would you explain this PIL error:
> > > >
> > > > File "/path/publisher/publisher.py", line 78, in handler
> > > > flow_instance.dispatch()
> > > > File "/path/scripts/updateprofile.py", line 92, in
> > > > dispatch
> > > > im.save(full, self.im_format)
> > > > File
> > > > "/usr/local/encap/Python-2.4.1/lib/python2.4/site-
> > > packages/PIL/Image.py",
> > > > line 1272, in save
> > > >
> > > > File
> > > > "/usr/local/encap/Python-2.4.1/lib/python2.4/site-
> > > packages/PIL/ImageFile.py"
> > > > , line 192, in load
> > > > IOError: image file is truncated (14432 bytes not processed)
> > > >
> > > > This only happened for large images, small ones uploaded fine.
> > > > The code is
> > > >
> > > > def dispatch(self):
> > > > pix = self.form['Filedata']
> > > > try:
> > > > im = Image.open(pix.file)
> > > > except IOError:
> > > > self.req.status = apache.HTTP_NOT_ACCEPTABLE
> > > > im.save(full, self.im_format)
> > > >
> > > > The error is in the last line. It did not occur with prefork.
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > > -----Original Message-----
> > > > > From: mod_python-bounces at modpython.org
> > > > [mailto:mod_python-
> > > > > bounces at modpython.org] On Behalf Of Martijn Moeling
> > > > > Sent: Sunday, November 04, 2007 5:16 AM
> > > > > To: Graham Dumpleton
> > > > > Cc: mod_python at modpython.org; Alec Matusis
> > > >
> > > > > Subject: Re: [mod_python] mod_python or apache scalability?
> > > > >
> > > > > hi,
> > > > >
> > > > > I use Pil without any problems, a I do with MySQLdb.
> > > > >
> > > > > I had similair problems however, they turned out to be MySQLdb
> > > > > related and not in a way you suspect
> > > > >
> > > > > Never ever use a global database connection variable. Create
> the
> > > > > database connection within your handler,
> > > > > end register a cleanup for closing it. I fixed a lot of trouble
> by
> > > > > altering my system by making all "global" stuff a method or
> > > property
> > > > > of the req object.
> > > > >
> > > > > The cleanup procedure cleans all objects which do anything to
> 3rd
> > > > > party external (net connected) stuff (like imap, MySQL)
> > > > >
> > > > >
> > > > > think of something like this:
> > > > >
> > > > >
> > > > > def handler_cleanup(req):
> > > > > req.db.cursor.close()
> > > > > req.db.close()
> > > > >
> > > > > Def handler (req):
> > > > > req.db=MySQLdb.connection(...........)
> > > > > req.cursor=req.db.cursor(...)
> > > > >
> > > > > ....
> > > > > ....
> > > > > ....
> > > > > req.register_cleanup(handler_cleanup,req)
> > > > > return apache.OK
> > > > >
> > > > > All my application specific code are classes, and instantated
> as a
> > > > > req.something and req is passed to all functions, which is neat
> > > since
> > > > > it is passed by reference meaning stacks stay small to
> > > > >
> > > > > If you use MySQLdb with mod_python use the above method, my
> > > > > production server is rock stable and handled over 1mln requests
> > > since
> > > > > last december without any reboots or problems.
> > > > >
> > > > > Martijn
> > > > >
> > > > > On Nov 4, 2007, at 11:48 AM, Graham Dumpleton wrote:
> > > > >
> > > > > > Hmmm, I do remember vaguely hearing questions about PIL
> thread
> > > safety
> > > > > > before, so it might be an issue. :-(
> > > > > >
> > > > > > Graham
> > > > > >
> > > > > > On 04/11/2007, Alec Matusis <matusis at matusis.com> wrote:
> > > > > >>> BTW, are you still using mod_python 3.1.4?
> > > > > >>
> > > > > >> No, we went to apache 2.2.6/ mod_python 3.3.1 combination.
> > > > > >>
> > > > > >> I will get MySQLdb to work thread-safely, but with PIL I am
> not
> > > so
> > > > > >> optimistic...
> > > > > >>
> > > > > >>> -----Original Message-----
> > > > > >>> From: Graham Dumpleton
> > > > [mailto:graham.dumpleton at gmail.com]
> > > > > >>> Sent: Sunday, November 04, 2007 2:33 AM
> > > > > >>> To: Alec Matusis
> > > > > >>> Cc: mod_python at modpython.org
> > > > > >>> Subject: Re: [mod_python] mod_python or apache scalability?
> > > > > >>>
> > > > > >>> BTW, are you still using mod_python 3.1.4? Older mod_python
> > > > > versions
> > > > > >>> have various bugs and you would be much better upgrading to
> > > 3.3.1
> > > > > if
> > > > > >>> you haven't already.
> > > > > >>>
> > > > > >>> Graham
> > > > > >>>
> > > > > >>> On 04/11/2007, Alec Matusis <matusis at matusis.com> wrote:
> > > > > >>>>> Which indicates that some third party package you are
> using
> > > is
> > > > > not
> > > > > >>>>> thread safe. This can occur if you are also using PHP as
> > > > > >>>>> various of
> > > > > >>>>> its third party packages are not thread safe.
> > > > > >>>>
> > > > > >>>> We are not using PHP, only python with mod_python.
> > > > > >>>> The only third party packages we use are MySQLdb adapter
> and
> > > PIL
> > > > > >>> image
> > > > > >>>> library.
> > > > > >>>>
> > > > > >>>> So I guess both PIL and MySQLdb have these problems.
> > > > > >>>>
> > > > > >>>>
> > > > > >>>>> -----Original Message-----
> > > > > >>>>> From: Graham Dumpleton
> > > > [mailto:graham.dumpleton at gmail.com]
> > > > > >>>>> Sent: Sunday, November 04, 2007 2:15 AM
> > > > > >>>>> To: Alec Matusis
> > > > > >>>>> Cc: mod_python at modpython.org
> > > > > >>>>> Subject: Re: [mod_python] mod_python or apache
> scalability?
> > > > > >>>>>
> > > > > >>>>> On 04/11/2007, Alec Matusis <matusis at matusis.com> wrote:
> > > > > >>>>>>> FWIW, I personally would try and move from prefork to
> > > worker
> > > > > >>> MPM as
> > > > > >>>>>>> the number of Apache child processes you are running
> with
> > > is to
> > > > > >>> my
> > > > > >>>>>>> mind excessive. Using worker would certainly drop
> memory
> > > usage
> > > > > >>> for
> > > > > >>>>> a
> > > > > >>>>>>> start as you wouldn't need as many child processes to
> be
> > > > > >>> started.
> > > > > >>>>>>
> > > > > >>>>>> I am just following up on this, since we tried worker
> MPM
> > > this
> > > > > >>>>> weekend.
> > > > > >>>>>> On our dev/stage it worked perfectly.
> > > > > >>>>>> On live, worker MPM freed up about 2GB of memory
> compared to
> > > > > >>> prefork.
> > > > > >>>>>> However, on live, it turned out to be unstable.
> > > > > >>>>>> This is what we say in the main error log:
> > > > > >>>>>>
> > > > > >>>>>> [Sun Nov 04 01:37:45 2007] [notice] child pid 20347 exit
> > > signal
> > > > > >>>>> Segmentation
> > > > > >>>>>> fault (11)
> > > > > >>>>>> [Sun Nov 04 01:37:45 2007] [notice] child pid 20460 exit
> > > signal
> > > > > >>>>> Aborted (6)
> > > > > >>>>>> [Sun Nov 04 01:37:45 2007] [notice] child pid 20515 exit
> > > signal
> > > > > >>>>> Segmentation
> > > > > >>>>>> fault (11)
> > > > > >>>>>> *** glibc detected *** double free or corruption
> (!prev):
> > > > > >>>>> 0x0000000000762c50
> > > > > >>>>>> ***
> > > > > >>>>>> *** glibc detected *** double free or corruption
> (!prev):
> > > > > >>>>> 0x000000000075d100
> > > > > >>>>>> ***
> > > > > >>>>>> [Sun Nov 04 01:37:46 2007] [notice] child pid 20152 exit
> > > signal
> > > > > >>>>> Segmentation
> > > > > >>>>>> fault (11)
> > > > > >>>>>>
> > > > > >>>>>> In the application log, we saw two types of errors:
> > > > > >>>>>>
> > > > > >>>>>> MySQLError: Connection to database failed
> > > > > >>>>>> OperationalError: (2006, 'MySQL server has gone away')
> > > > > >>>>>>
> > > > > >>>>>> and
> > > > > >>>>>>
> > > > > >>>>>> IOError: image file is truncated (44 bytes not
> processed)
> > > > > >>>>>>
> > > > > >>>>>> The first type has to do with MySQLdb module, but the
> second
> > > one
> > > > > >>>>> occurred
> > > > > >>>>>> when large images were uploaded.
> > > > > >>>>>
> > > > > >>>>> Which indicates that some third party package you are
> using
> > > is
> > > > > not
> > > > > >>>>> thread safe. This can occur if you are also using PHP as
> > > > > >>>>> various of
> > > > > >>>>> its third party packages are not thread safe. Also ensure
> > > that
> > > > > you
> > > > > >>> are
> > > > > >>>>> using the latest available Python database adapters and
> that
> > > they
> > > > > >>> are
> > > > > >>>>> compiled against thread safe reentrant libraries.
> > > > > >>>>>
> > > > > >>>>> Graham
> > > > > >>>>>
> > > > > >>>>>> We had to revert to prefork as a result of this.
> > > > > >>>>>>
> > > > > >>>>>> On another note, I managed to empirically find the
> maximum
> > > > > >>>>> ServerLimit for
> > > > > >>>>>> prefork, before the machine dies from swapping.
> > > > > >>>>>> It is 380 with 4GB RAM.
> > > > > >>>>>>
> > > > > >>>>>>> -----Original Message-----
> > > > > >>>>>>> From: Graham Dumpleton
> > > > [mailto:graham.dumpleton at gmail.com]
> > > > > >>>>>>> Sent: Monday, October 01, 2007 6:47 PM
> > > > > >>>>>>> To: Alec Matusis
> > > > > >>>>>>> Cc: mod_python at modpython.org
> > > > > >>>>>>> Subject: Re: [mod_python] mod_python or apache
> scalability?
> > > > > >>>>>>>
> > > > > >>>>>>> On 01/10/2007, Alec Matusis <matusis at matusis.com>
> wrote:
> > > > > >>>>>>>> in the apache error log. We also got
> > > > > >>>>>>>>
> > > > > >>>>>>>> kernel: possible SYN flooding on port 80. Sending
> cookies.
> > > > > >>>>>>>>
> > > > > >>>>>>>> in /var/log/messages system log.
> > > > > >>>>>>>
> > > > > >>>>>>> Have you determined for certain that you aren't the
> target
> > > of
> > > > > >>> an
> > > > > >>>>>>> external SYN Flood DOS attack?
> > > > > >>>>>>>
> > > > > >>>>>>> Do a Google search for 'kernel: possible SYN flooding
> on
> > > port
> > > > > >>> 80.
> > > > > >>>>>>> Sending cookies' and you will find lots of stuff to
> read.
> > > Your
> > > > > >>>>> running
> > > > > >>>>>>> out of or having a large number of socket connections
> may
> > > be
> > > > > >>>>>>> symptomatic of a large number of half open connections
> > > being
> > > > > >>>>> created
> > > > > >>>>>>> and then being left in TIME_WAIT. Thus perhaps do some
> > > better
> > > > > >>>>> analysis
> > > > > >>>>>>> of socket connection states using netstat. If not a SYN
> > > Flood,
> > > > > >>> then
> > > > > >>>>>>> possibly follow some of the other suggestions in the
> pages
> > > you
> > > > > >>> will
> > > > > >>>>>>> find when you do the search.
> > > > > >>>>>>>
> > > > > >>>>>>> FWIW, I personally would try and move from prefork to
> > > worker
> > > > > >>> MPM as
> > > > > >>>>>>> the number of Apache child processes you are running
> with
> > > is to
> > > > > >>> my
> > > > > >>>>>>> mind excessive. Using worker would certainly drop
> memory
> > > usage
> > > > > >>> for
> > > > > >>>>> a
> > > > > >>>>>>> start as you wouldn't need as many child processes to
> be
> > > > > >>> started. I
> > > > > >>>>>>> wouldn't be concerned about running out of threads as
> when
> > > > > >>> running
> > > > > >>>>>>> worker I wouldn't suggest more than 25 threads per
> process
> > > as a
> > > > > >>>>>>> starting point anyway. If your mod_python application
> was
> > > > > >>> creating
> > > > > >>>>>>> lots of threads, you are likely to hit the thread limit
> > > with
> > > > > >>>>> prefork
> > > > > >>>>>>> and not just worker so which MPM is used shouldn't be
> an
> > > issue
> > > > > >>> in
> > > > > >>>>> that
> > > > > >>>>>>> case.
> > > > > >>>>>>>
> > > > > >>>>>>> BTW, what operating system are you using?
> > > > > >>>>>>>
> > > > > >>>>>>> Graham
> > > > > >>>>>>
> > > > > >>>>>>
> > > > > >>>>
> > > > > >>>>
> > > > > >>
> > > > > >>
> > > > > > _______________________________________________
> > > > > > Mod_python mailing list
> > > > > > Mod_python at modpython.org
> > > > > >
> > > > http://mailman.modpython.org/mailman/listinfo/mod_python
> > > > >
> > > > > _______________________________________________
> > > > > Mod_python mailing list
> > > > > Mod_python at modpython.org
> > > > > http://mailman.modpython.org/mailman/listinfo/mod_python
> > > >
> > > >
> > > >
> > > >
> > > > _______________________________________________
> > > > Mod_python mailing list
> > > > Mod_python at modpython.org
> > > > http://mailman.modpython.org/mailman/listinfo/mod_python
> > > >
> > > >
> > > _______________________________________________
> > > Mod_python mailing list
> > > Mod_python at modpython.org
> > > http://mailman.modpython.org/mailman/listinfo/mod_python
> >
> >
|