[mod_python] mod_python or apache scalability?

Hmmm, I do remember vaguely hearing questions about PIL thread safety
before, so it might be an issue. :-(

Graham

On 04/11/2007, Alec Matusis <matusis at matusis.com> wrote:
> > BTW, are you still using mod_python 3.1.4?
>
> No, we went to apache 2.2.6/ mod_python 3.3.1 combination.
>
> I will get MySQLdb to work thread-safely, but with PIL I am not so
> optimistic...
>
> > -----Original Message-----
> > From: Graham Dumpleton [mailto:graham.dumpleton at gmail.com]
> > Sent: Sunday, November 04, 2007 2:33 AM
> > To: Alec Matusis
> > Cc: mod_python at modpython.org
> > Subject: Re: [mod_python] mod_python or apache scalability?
> >
> > BTW, are you still using mod_python 3.1.4? Older mod_python versions
> > have various bugs and you would be much better upgrading to 3.3.1 if
> > you haven't already.
> >
> > Graham
> >
> > On 04/11/2007, Alec Matusis <matusis at matusis.com> wrote:
> > > > Which indicates that some third party package you are using is not
> > > > thread safe. This can occur if you are also using PHP as various of
> > > > its third party packages are not thread safe.
> > >
> > > We are not using PHP, only python with mod_python.
> > > The only third party packages we use are MySQLdb adapter and PIL
> > image
> > > library.
> > >
> > > So I guess both PIL and MySQLdb have these problems.
> > >
> > >
> > > > -----Original Message-----
> > > > From: Graham Dumpleton [mailto:graham.dumpleton at gmail.com]
> > > > Sent: Sunday, November 04, 2007 2:15 AM
> > > > To: Alec Matusis
> > > > Cc: mod_python at modpython.org
> > > > Subject: Re: [mod_python] mod_python or apache scalability?
> > > >
> > > > On 04/11/2007, Alec Matusis <matusis at matusis.com> wrote:
> > > > > > FWIW, I personally would try and move from prefork to worker
> > MPM as
> > > > > > the number of Apache child processes you are running with is to
> > my
> > > > > > mind excessive. Using worker would certainly drop memory usage
> > for
> > > > a
> > > > > > start as you wouldn't need as many child processes to be
> > started.
> > > > >
> > > > > I am just following up on this, since we tried worker MPM this
> > > > weekend.
> > > > > On our dev/stage it worked perfectly.
> > > > > On live, worker MPM freed up about 2GB of memory compared to
> > prefork.
> > > > > However, on live, it turned out to be unstable.
> > > > > This is what we say in the main error log:
> > > > >
> > > > > [Sun Nov 04 01:37:45 2007] [notice] child pid 20347 exit signal
> > > > Segmentation
> > > > > fault (11)
> > > > > [Sun Nov 04 01:37:45 2007] [notice] child pid 20460 exit signal
> > > > Aborted (6)
> > > > > [Sun Nov 04 01:37:45 2007] [notice] child pid 20515 exit signal
> > > > Segmentation
> > > > > fault (11)
> > > > > *** glibc detected *** double free or corruption (!prev):
> > > > 0x0000000000762c50
> > > > > ***
> > > > > *** glibc detected *** double free or corruption (!prev):
> > > > 0x000000000075d100
> > > > > ***
> > > > > [Sun Nov 04 01:37:46 2007] [notice] child pid 20152 exit signal
> > > > Segmentation
> > > > > fault (11)
> > > > >
> > > > > In the application log, we saw two types of errors:
> > > > >
> > > > > MySQLError: Connection to database failed
> > > > > OperationalError: (2006, 'MySQL server has gone away')
> > > > >
> > > > > and
> > > > >
> > > > > IOError: image file is truncated (44 bytes not processed)
> > > > >
> > > > > The first type has to do with MySQLdb module, but the second one
> > > > occurred
> > > > > when large images were uploaded.
> > > >
> > > > Which indicates that some third party package you are using is not
> > > > thread safe. This can occur if you are also using PHP as various of
> > > > its third party packages are not thread safe. Also ensure that you
> > are
> > > > using the latest available Python database adapters and that they
> > are
> > > > compiled against thread safe reentrant libraries.
> > > >
> > > > Graham
> > > >
> > > > > We had to revert to prefork as a result of this.
> > > > >
> > > > > On another note, I managed to empirically find the maximum
> > > > ServerLimit for
> > > > > prefork, before the machine dies from swapping.
> > > > > It is 380 with 4GB RAM.
> > > > >
> > > > > > -----Original Message-----
> > > > > > From: Graham Dumpleton [mailto:graham.dumpleton at gmail.com]
> > > > > > Sent: Monday, October 01, 2007 6:47 PM
> > > > > > To: Alec Matusis
> > > > > > Cc: mod_python at modpython.org
> > > > > > Subject: Re: [mod_python] mod_python or apache scalability?
> > > > > >
> > > > > > On 01/10/2007, Alec Matusis <matusis at matusis.com> wrote:
> > > > > > > in the apache error log. We also got
> > > > > > >
> > > > > > > kernel: possible SYN flooding on port 80. Sending cookies.
> > > > > > >
> > > > > > > in /var/log/messages system log.
> > > > > >
> > > > > > Have you determined for certain that you aren't the target of
> > an
> > > > > > external SYN Flood DOS attack?
> > > > > >
> > > > > > Do a Google search for 'kernel: possible SYN flooding on port
> > 80.
> > > > > > Sending cookies' and you will find lots of stuff to read. Your
> > > > running
> > > > > > out of or having a large number of socket connections may be
> > > > > > symptomatic of a large number of half open connections being
> > > > created
> > > > > > and then being left in TIME_WAIT. Thus perhaps do some better
> > > > analysis
> > > > > > of socket connection states using netstat. If not a SYN Flood,
> > then
> > > > > > possibly follow some of the other suggestions in the pages you
> > will
> > > > > > find when you do the search.
> > > > > >
> > > > > > FWIW, I personally would try and move from prefork to worker
> > MPM as
> > > > > > the number of Apache child processes you are running with is to
> > my
> > > > > > mind excessive. Using worker would certainly drop memory usage
> > for
> > > > a
> > > > > > start as you wouldn't need as many child processes to be
> > started. I
> > > > > > wouldn't be concerned about running out of threads as when
> > running
> > > > > > worker I wouldn't suggest more than 25 threads per process as a
> > > > > > starting point anyway. If your mod_python application was
> > creating
> > > > > > lots of threads, you are likely to hit the thread limit with
> > > > prefork
> > > > > > and not just worker so which MPM is used shouldn't be an issue
> > in
> > > > that
> > > > > > case.
> > > > > >
> > > > > > BTW, what operating system are you using?
> > > > > >
> > > > > > Graham
> > > > >
> > > > >
> > >
> > >
>
>