[mod_python] mod_python or apache scalability?

BTW, are you still using mod_python 3.1.4? Older mod_python versions
have various bugs and you would be much better upgrading to 3.3.1 if
you haven't already.

Graham

On 04/11/2007, Alec Matusis <matusis at matusis.com> wrote:
> > Which indicates that some third party package you are using is not
> > thread safe. This can occur if you are also using PHP as various of
> > its third party packages are not thread safe.
>
> We are not using PHP, only python with mod_python.
> The only third party packages we use are MySQLdb adapter and PIL image
> library.
>
> So I guess both PIL and MySQLdb have these problems.
>
>
> > -----Original Message-----
> > From: Graham Dumpleton [mailto:graham.dumpleton at gmail.com]
> > Sent: Sunday, November 04, 2007 2:15 AM
> > To: Alec Matusis
> > Cc: mod_python at modpython.org
> > Subject: Re: [mod_python] mod_python or apache scalability?
> >
> > On 04/11/2007, Alec Matusis <matusis at matusis.com> wrote:
> > > > FWIW, I personally would try and move from prefork to worker MPM as
> > > > the number of Apache child processes you are running with is to my
> > > > mind excessive. Using worker would certainly drop memory usage for
> > a
> > > > start as you wouldn't need as many child processes to be started.
> > >
> > > I am just following up on this, since we tried worker MPM this
> > weekend.
> > > On our dev/stage it worked perfectly.
> > > On live, worker MPM freed up about 2GB of memory compared to prefork.
> > > However, on live, it turned out to be unstable.
> > > This is what we say in the main error log:
> > >
> > > [Sun Nov 04 01:37:45 2007] [notice] child pid 20347 exit signal
> > Segmentation
> > > fault (11)
> > > [Sun Nov 04 01:37:45 2007] [notice] child pid 20460 exit signal
> > Aborted (6)
> > > [Sun Nov 04 01:37:45 2007] [notice] child pid 20515 exit signal
> > Segmentation
> > > fault (11)
> > > *** glibc detected *** double free or corruption (!prev):
> > 0x0000000000762c50
> > > ***
> > > *** glibc detected *** double free or corruption (!prev):
> > 0x000000000075d100
> > > ***
> > > [Sun Nov 04 01:37:46 2007] [notice] child pid 20152 exit signal
> > Segmentation
> > > fault (11)
> > >
> > > In the application log, we saw two types of errors:
> > >
> > > MySQLError: Connection to database failed
> > > OperationalError: (2006, 'MySQL server has gone away')
> > >
> > > and
> > >
> > > IOError: image file is truncated (44 bytes not processed)
> > >
> > > The first type has to do with MySQLdb module, but the second one
> > occurred
> > > when large images were uploaded.
> >
> > Which indicates that some third party package you are using is not
> > thread safe. This can occur if you are also using PHP as various of
> > its third party packages are not thread safe. Also ensure that you are
> > using the latest available Python database adapters and that they are
> > compiled against thread safe reentrant libraries.
> >
> > Graham
> >
> > > We had to revert to prefork as a result of this.
> > >
> > > On another note, I managed to empirically find the maximum
> > ServerLimit for
> > > prefork, before the machine dies from swapping.
> > > It is 380 with 4GB RAM.
> > >
> > > > -----Original Message-----
> > > > From: Graham Dumpleton [mailto:graham.dumpleton at gmail.com]
> > > > Sent: Monday, October 01, 2007 6:47 PM
> > > > To: Alec Matusis
> > > > Cc: mod_python at modpython.org
> > > > Subject: Re: [mod_python] mod_python or apache scalability?
> > > >
> > > > On 01/10/2007, Alec Matusis <matusis at matusis.com> wrote:
> > > > > in the apache error log. We also got
> > > > >
> > > > > kernel: possible SYN flooding on port 80. Sending cookies.
> > > > >
> > > > > in /var/log/messages system log.
> > > >
> > > > Have you determined for certain that you aren't the target of an
> > > > external SYN Flood DOS attack?
> > > >
> > > > Do a Google search for 'kernel: possible SYN flooding on port 80.
> > > > Sending cookies' and you will find lots of stuff to read. Your
> > running
> > > > out of or having a large number of socket connections may be
> > > > symptomatic of a large number of half open connections being
> > created
> > > > and then being left in TIME_WAIT. Thus perhaps do some better
> > analysis
> > > > of socket connection states using netstat. If not a SYN Flood, then
> > > > possibly follow some of the other suggestions in the pages you will
> > > > find when you do the search.
> > > >
> > > > FWIW, I personally would try and move from prefork to worker MPM as
> > > > the number of Apache child processes you are running with is to my
> > > > mind excessive. Using worker would certainly drop memory usage for
> > a
> > > > start as you wouldn't need as many child processes to be started. I
> > > > wouldn't be concerned about running out of threads as when running
> > > > worker I wouldn't suggest more than 25 threads per process as a
> > > > starting point anyway. If your mod_python application was creating
> > > > lots of threads, you are likely to hit the thread limit with
> > prefork
> > > > and not just worker so which MPM is used shouldn't be an issue in
> > that
> > > > case.
> > > >
> > > > BTW, what operating system are you using?
> > > >
> > > > Graham
> > >
> > >
>
>