[mod_python] mod_python or apache scalability?

On 04/11/2007, Alec Matusis <matusis at matusis.com> wrote:
> > FWIW, I personally would try and move from prefork to worker MPM as
> > the number of Apache child processes you are running with is to my
> > mind excessive. Using worker would certainly drop memory usage for a
> > start as you wouldn't need as many child processes to be started.
>
> I am just following up on this, since we tried worker MPM this weekend.
> On our dev/stage it worked perfectly.
> On live, worker MPM freed up about 2GB of memory compared to prefork.
> However, on live, it turned out to be unstable.
> This is what we say in the main error log:
>
> [Sun Nov 04 01:37:45 2007] [notice] child pid 20347 exit signal Segmentation
> fault (11)
> [Sun Nov 04 01:37:45 2007] [notice] child pid 20460 exit signal Aborted (6)
> [Sun Nov 04 01:37:45 2007] [notice] child pid 20515 exit signal Segmentation
> fault (11)
> *** glibc detected *** double free or corruption (!prev): 0x0000000000762c50
> ***
> *** glibc detected *** double free or corruption (!prev): 0x000000000075d100
> ***
> [Sun Nov 04 01:37:46 2007] [notice] child pid 20152 exit signal Segmentation
> fault (11)
>
> In the application log, we saw two types of errors:
>
> MySQLError: Connection to database failed
> OperationalError: (2006, 'MySQL server has gone away')
>
> and
>
> IOError: image file is truncated (44 bytes not processed)
>
> The first type has to do with MySQLdb module, but the second one occurred
> when large images were uploaded.

Which indicates that some third party package you are using is not
thread safe. This can occur if you are also using PHP as various of
its third party packages are not thread safe. Also ensure that you are
using the latest available Python database adapters and that they are
compiled against thread safe reentrant libraries.

Graham

> We had to revert to prefork as a result of this.
>
> On another note, I managed to empirically find the maximum ServerLimit for
> prefork, before the machine dies from swapping.
> It is 380 with 4GB RAM.
>
> > -----Original Message-----
> > From: Graham Dumpleton [mailto:graham.dumpleton at gmail.com]
> > Sent: Monday, October 01, 2007 6:47 PM
> > To: Alec Matusis
> > Cc: mod_python at modpython.org
> > Subject: Re: [mod_python] mod_python or apache scalability?
> >
> > On 01/10/2007, Alec Matusis <matusis at matusis.com> wrote:
> > > in the apache error log. We also got
> > >
> > > kernel: possible SYN flooding on port 80. Sending cookies.
> > >
> > > in /var/log/messages system log.
> >
> > Have you determined for certain that you aren't the target of an
> > external SYN Flood DOS attack?
> >
> > Do a Google search for 'kernel: possible SYN flooding on port 80.
> > Sending cookies' and you will find lots of stuff to read. Your running
> > out of or having a large number of socket connections may be
> > symptomatic of a large number of half open connections being created
> > and then being left in TIME_WAIT. Thus perhaps do some better analysis
> > of socket connection states using netstat. If not a SYN Flood, then
> > possibly follow some of the other suggestions in the pages you will
> > find when you do the search.
> >
> > FWIW, I personally would try and move from prefork to worker MPM as
> > the number of Apache child processes you are running with is to my
> > mind excessive. Using worker would certainly drop memory usage for a
> > start as you wouldn't need as many child processes to be started. I
> > wouldn't be concerned about running out of threads as when running
> > worker I wouldn't suggest more than 25 threads per process as a
> > starting point anyway. If your mod_python application was creating
> > lots of threads, you are likely to hit the thread limit with prefork
> > and not just worker so which MPM is used shouldn't be an issue in that
> > case.
> >
> > BTW, what operating system are you using?
> >
> > Graham
>
>