[mod_python] mod_python or apache scalability?

In order so we can understand things better ...

1. Please explain what application you are running on top of
mod_python. Show the Apache configuration applying to the application
you are running on top of mod_python.

2. Explain why you are running prefork and not worker MPM. Are you
also running some PHP application that will not work with worker MPM?

3. Explain what the relationship is between your mod_python
application and your memory hogging twisted back end processes, ie.,
do they communicate with each other and how.

4. Indicate how big your Apache and twisted processes are growing.

5. Indicate whether you have used the netstat program to try and
determine how many socket connections are being held open by Apache
and twisted processes.

6. Indicate whether you have tried changing the Apache configuration
for how many keep alive connections are maintained and how long keep
alive connections are kept open. For some guidance on these see recent
blog post at:

  http://lucumr.pocoo.org/cogitations/2007/09/30/pushing-apache-performance

7. Indicate what range of values you have experimented with for
MaxRequestsPerChild.

At the moment the description of your problem is a bit vague so all
the above detail would help immensely as far as us being able to given
any ideas.

Please try not to gloss over details, the more details you give the
more helpful we might be able to be. For example, you don't even
mention that your system is shared with some very large twisted
processes. I only know that this might be the case from a short
followup you made to HTTPD dev list. When you say twisted, I presume
though you mean Python Twisted framework.

Graham

On 01/10/2007, Alec Matusis <matusis at matusis.com> wrote:
> I am sorry in advance if this turns out to be an apache-related issue, but
> when I posted this on apache list, it has been suggested that it might be an
> application issue, so I am reposting it here.
>
>
> We are running a busy mod_python/3.1.4 Python/2.4.1 server on 2.6.9 kernel,
> that suddenly becomes very slow- requests either time out, or it takes
> 10-20sec to serve a 1K thumbnail.
> It is somewhat correlated with load spikes, but not perfectly (by looking at
> the bandwidth graph, it never happens during the low bandwidth periods at
> night, but it does not coincide with peaks of b/w)
>
> When we initially encountered an apache overload, it was always accompanied
> with
>
> [error] server reached MaxClients setting, consider raising the MaxClients
> setting
>
> in the apache error log. We also got
>
> kernel: possible SYN flooding on port 80. Sending cookies.
>
> in /var/log/messages system log.
>
> After that I raised MaxClients from 200 to 300. The problem initially
> disappeared, but after our bandwidth grew a bit more, we got this behavior
> again.
> Now apache crashes (becomes very slow) silently, with no warning in apache
> error logs at all (although we still get SYN flood message in the system
> log)
> When apache is this 'slow' regime, /server-status still shows available
> slots, i.e. MaxClients is not reached.
>
> This is the relevant part of httpd.conf:
>
> ServerLimit 300
> # we are using prefork MPM
> StartServers 10
> MinSpareServers 5
> MaxSpareServers 20
> MaxClients 300
> MaxRequestsPerChild 10000
> MaxMemFree 2500
>
> The server has 4GB of physical RAM and 4GB of swap. During these apache
> "slowdowns", the swap size is still 0 and vmstat shows no swapping at all.
> I suspect the problem may be in
>
> MaxMemFree 2500
>
> but then I would expect some kind of"out of memory" errors in the logs?
>
> I am posting it on this list since I have not gotten a response in the users
> list, and I think it's a bug for two reasons:
>
> 1) When apache is in this slow "degraded" regime, I would expect a log
> message in the apache error log, with an explanation why.
>
> 3) If this is related to resource exhaustion, I would expect the server to
> recover from this regime by itself when the load subsides, but this is not
> the case. Only apachectl start/stop recovers the server.
>
>
>
> _______________________________________________
> Mod_python mailing list
> Mod_python at modpython.org
> http://mailman.modpython.org/mailman/listinfo/mod_python
>