[mod_python] mod_python or apache scalability?

Hmm it looks to me that you have an upload limit in your apache config.
 
It might be the LimitRequestBody directive.
 
Another thing might be that your processing is faster than the data comes in, but I doubt that
Take a close look at you apache conf, and your apache error log (right after an upload or with tail -f /var/log/httpd/error_log (or where ever your logdir is)).
 
Martijn
 

________________________________

Van: Alec Matusis [mailto:matusis at matusis.com]
Verzonden: zo 04.11.2007 21:46
Aan: Martijn Moeling
CC: mod_python at modpython.org
Onderwerp: RE: [mod_python] mod_python or apache scalability?



> I had similair problems however, they turned out to be MySQLdb
> related and not in a way you suspect

Thanks! I will try separating MySQLdb connections. We do pass the connection
around currently.

However, how would you explain this PIL error:

    File "/path/publisher/publisher.py", line 78, in handler
 flow_instance.dispatch()
 File "/path/scripts/updateprofile.py", line 92, in dispatch
 im.save(full, self.im_format)
    File
"/usr/local/encap/Python-2.4.1/lib/python2.4/site-packages/PIL/Image.py",
line 1272, in save

 File
"/usr/local/encap/Python-2.4.1/lib/python2.4/site-packages/PIL/ImageFile.py"
, line 192, in load
 IOError: image file is truncated (14432 bytes not processed)

This only happened for large images, small ones uploaded fine.
The code is

def dispatch(self):
    pix = self.form['Filedata']
    try:
        im = Image.open(pix.file)
    except IOError:
        self.req.status = apache.HTTP_NOT_ACCEPTABLE
    im.save(full, self.im_format)

The error is in the last line. It did not occur with prefork.
   




> -----Original Message-----
> From: mod_python-bounces at modpython.org [mailto:mod_python-
> bounces at modpython.org] On Behalf Of Martijn Moeling
> Sent: Sunday, November 04, 2007 5:16 AM
> To: Graham Dumpleton
> Cc: mod_python at modpython.org; Alec Matusis
> Subject: Re: [mod_python] mod_python or apache scalability?
>
> hi,
>
> I use Pil without any problems, a I do with MySQLdb.
>
> I had similair problems however, they turned out to be MySQLdb
> related and not in a way you suspect
>
> Never ever use a global database connection variable. Create the
> database connection within your handler,
> end register a cleanup for closing it. I fixed a lot of trouble by
> altering my system by making all "global" stuff a method or property
> of the req object.
>
> The cleanup procedure cleans all objects which do anything to 3rd
> party external (net connected) stuff (like imap, MySQL)
>
>
> think of something like this:
>
>
> def handler_cleanup(req):
>       req.db.cursor.close()
>       req.db.close()
>
> Def handler (req):
>       req.db=MySQLdb.connection(...........)
>       req.cursor=req.db.cursor(...)
>
>       ....
>       ....
>       ....
>       req.register_cleanup(handler_cleanup,req)
>       return apache.OK
>
> All my application specific code are classes, and instantated as a
> req.something and req is passed to all functions, which is neat since
> it is passed by reference meaning stacks stay small to
>
> If you use MySQLdb with mod_python use the above method, my
> production server is rock stable and handled over 1mln requests since
> last december without any reboots or problems.
>
> Martijn
>
> On Nov 4, 2007, at 11:48 AM, Graham Dumpleton wrote:
>
> > Hmmm, I do remember vaguely hearing questions about PIL thread safety
> > before, so it might be an issue. :-(
> >
> > Graham
> >
> > On 04/11/2007, Alec Matusis <matusis at matusis.com> wrote:
> >>> BTW, are you still using mod_python 3.1.4?
> >>
> >> No, we went to apache 2.2.6/ mod_python 3.3.1 combination.
> >>
> >> I will get MySQLdb to work thread-safely, but with PIL I am not so
> >> optimistic...
> >>
> >>> -----Original Message-----
> >>> From: Graham Dumpleton [mailto:graham.dumpleton at gmail.com]
> >>> Sent: Sunday, November 04, 2007 2:33 AM
> >>> To: Alec Matusis
> >>> Cc: mod_python at modpython.org
> >>> Subject: Re: [mod_python] mod_python or apache scalability?
> >>>
> >>> BTW, are you still using mod_python 3.1.4? Older mod_python
> versions
> >>> have various bugs and you would be much better upgrading to 3.3.1
> if
> >>> you haven't already.
> >>>
> >>> Graham
> >>>
> >>> On 04/11/2007, Alec Matusis <matusis at matusis.com> wrote:
> >>>>> Which indicates that some third party package you are using is
> not
> >>>>> thread safe. This can occur if you are also using PHP as
> >>>>> various of
> >>>>> its third party packages are not thread safe.
> >>>>
> >>>> We are not using PHP, only python with mod_python.
> >>>> The only third party packages we use are MySQLdb adapter and PIL
> >>> image
> >>>> library.
> >>>>
> >>>> So I guess both PIL and MySQLdb have these problems.
> >>>>
> >>>>
> >>>>> -----Original Message-----
> >>>>> From: Graham Dumpleton [mailto:graham.dumpleton at gmail.com]
> >>>>> Sent: Sunday, November 04, 2007 2:15 AM
> >>>>> To: Alec Matusis
> >>>>> Cc: mod_python at modpython.org
> >>>>> Subject: Re: [mod_python] mod_python or apache scalability?
> >>>>>
> >>>>> On 04/11/2007, Alec Matusis <matusis at matusis.com> wrote:
> >>>>>>> FWIW, I personally would try and move from prefork to worker
> >>> MPM as
> >>>>>>> the number of Apache child processes you are running with is to
> >>> my
> >>>>>>> mind excessive. Using worker would certainly drop memory usage
> >>> for
> >>>>> a
> >>>>>>> start as you wouldn't need as many child processes to be
> >>> started.
> >>>>>>
> >>>>>> I am just following up on this, since we tried worker MPM this
> >>>>> weekend.
> >>>>>> On our dev/stage it worked perfectly.
> >>>>>> On live, worker MPM freed up about 2GB of memory compared to
> >>> prefork.
> >>>>>> However, on live, it turned out to be unstable.
> >>>>>> This is what we say in the main error log:
> >>>>>>
> >>>>>> [Sun Nov 04 01:37:45 2007] [notice] child pid 20347 exit signal
> >>>>> Segmentation
> >>>>>> fault (11)
> >>>>>> [Sun Nov 04 01:37:45 2007] [notice] child pid 20460 exit signal
> >>>>> Aborted (6)
> >>>>>> [Sun Nov 04 01:37:45 2007] [notice] child pid 20515 exit signal
> >>>>> Segmentation
> >>>>>> fault (11)
> >>>>>> *** glibc detected *** double free or corruption (!prev):
> >>>>> 0x0000000000762c50
> >>>>>> ***
> >>>>>> *** glibc detected *** double free or corruption (!prev):
> >>>>> 0x000000000075d100
> >>>>>> ***
> >>>>>> [Sun Nov 04 01:37:46 2007] [notice] child pid 20152 exit signal
> >>>>> Segmentation
> >>>>>> fault (11)
> >>>>>>
> >>>>>> In the application log, we saw two types of errors:
> >>>>>>
> >>>>>> MySQLError: Connection to database failed
> >>>>>> OperationalError: (2006, 'MySQL server has gone away')
> >>>>>>
> >>>>>> and
> >>>>>>
> >>>>>> IOError: image file is truncated (44 bytes not processed)
> >>>>>>
> >>>>>> The first type has to do with MySQLdb module, but the second one
> >>>>> occurred
> >>>>>> when large images were uploaded.
> >>>>>
> >>>>> Which indicates that some third party package you are using is
> not
> >>>>> thread safe. This can occur if you are also using PHP as
> >>>>> various of
> >>>>> its third party packages are not thread safe. Also ensure that
> you
> >>> are
> >>>>> using the latest available Python database adapters and that they
> >>> are
> >>>>> compiled against thread safe reentrant libraries.
> >>>>>
> >>>>> Graham
> >>>>>
> >>>>>> We had to revert to prefork as a result of this.
> >>>>>>
> >>>>>> On another note, I managed to empirically find the maximum
> >>>>> ServerLimit for
> >>>>>> prefork, before the machine dies from swapping.
> >>>>>> It is 380 with 4GB RAM.
> >>>>>>
> >>>>>>> -----Original Message-----
> >>>>>>> From: Graham Dumpleton [mailto:graham.dumpleton at gmail.com]
> >>>>>>> Sent: Monday, October 01, 2007 6:47 PM
> >>>>>>> To: Alec Matusis
> >>>>>>> Cc: mod_python at modpython.org
> >>>>>>> Subject: Re: [mod_python] mod_python or apache scalability?
> >>>>>>>
> >>>>>>> On 01/10/2007, Alec Matusis <matusis at matusis.com> wrote:
> >>>>>>>> in the apache error log. We also got
> >>>>>>>>
> >>>>>>>> kernel: possible SYN flooding on port 80. Sending cookies.
> >>>>>>>>
> >>>>>>>> in /var/log/messages system log.
> >>>>>>>
> >>>>>>> Have you determined for certain that you aren't the target of
> >>> an
> >>>>>>> external SYN Flood DOS attack?
> >>>>>>>
> >>>>>>> Do a Google search for 'kernel: possible SYN flooding on port
> >>> 80.
> >>>>>>> Sending cookies' and you will find lots of stuff to read. Your
> >>>>> running
> >>>>>>> out of or having a large number of socket connections may be
> >>>>>>> symptomatic of a large number of half open connections being
> >>>>> created
> >>>>>>> and then being left in TIME_WAIT. Thus perhaps do some better
> >>>>> analysis
> >>>>>>> of socket connection states using netstat. If not a SYN Flood,
> >>> then
> >>>>>>> possibly follow some of the other suggestions in the pages you
> >>> will
> >>>>>>> find when you do the search.
> >>>>>>>
> >>>>>>> FWIW, I personally would try and move from prefork to worker
> >>> MPM as
> >>>>>>> the number of Apache child processes you are running with is to
> >>> my
> >>>>>>> mind excessive. Using worker would certainly drop memory usage
> >>> for
> >>>>> a
> >>>>>>> start as you wouldn't need as many child processes to be
> >>> started. I
> >>>>>>> wouldn't be concerned about running out of threads as when
> >>> running
> >>>>>>> worker I wouldn't suggest more than 25 threads per process as a
> >>>>>>> starting point anyway. If your mod_python application was
> >>> creating
> >>>>>>> lots of threads, you are likely to hit the thread limit with
> >>>>> prefork
> >>>>>>> and not just worker so which MPM is used shouldn't be an issue
> >>> in
> >>>>> that
> >>>>>>> case.
> >>>>>>>
> >>>>>>> BTW, what operating system are you using?
> >>>>>>>
> >>>>>>> Graham
> >>>>>>
> >>>>>>
> >>>>
> >>>>
> >>
> >>
> > _______________________________________________
> > Mod_python mailing list
> > Mod_python at modpython.org
> > http://mailman.modpython.org/mailman/listinfo/mod_python
>
> _______________________________________________
> Mod_python mailing list
> Mod_python at modpython.org
> http://mailman.modpython.org/mailman/listinfo/mod_python



-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mm_cfg_has_not_been_edited_to_set_host_domains/pipermail/mod_python/attachments/20071105/2ccc7158/attachment.html