[mod_python] High traffic recommendations

Marc D. Murray wrote:
>>What did your custom Session code do ? Can you post it, so that we try
>>and see what could cause the problem ?
> 
> The intention was to prevent multiple users from sharing username and
> password. And to lock an account if it was used from too many
> locations. 
> 
> I'll post it, but before we go there, I should tell you that, I'm still
> getting that dreaded "MaxClients reached" error again. Taking somewhere
> of of the order of 6hrs now though.

But you are still using sessions, right? There are 2 types of locks used 
in DbmSession, a global lock for serializing access to the underlying 
dbm file, and then an additional number of locks that are used for 
locking individual sessions. (The exact number in 3.1.4 is determined at 
run time, and for your config is likely 32. You'll see the actual number 
on startup in the error log. There is configuration option for compiling 
3.2.7 which allows you to set the total number of locks. We are 
investigating implementing a new apache directive which would allow the 
number of locks to be specified at runtime). Since access to the dbm 
file is serialized, you could be experiencing a bottleneck when your 
server is under load.

How many sessions objects are you saving? Performance drops when 
inserting a record into an already large dbm database. I did some 
benchmarks when I was writing the FileSession class. I think you might 
find some of the data interesting:
http://www.mail-archive.com/python-dev@httpd.apache.org/msg00136.html

Another performance killer with DbmSessions is the expired session 
cleanup. Each record in the database must be unpickled and examined to 
see if it is expired and should be deleted. For a large number of 
session objects this can be really time consuming. The cleanup is kicked 
off after a random number of requests (approx 1 in 1000), but for a 
heavily loaded site this could mean that a cleanup is almost always 
running. During this time the dbm file is locked, so all other requests 
which are using sessions will be blocked. This means that Apache will be 
queue requests and could conceivably hit the MaxClients.

These are the sort of issues that motivated the creation of FileSession, 
which gives a much finer control over the cleanup process, and avoids 
the global lock problem completely.

> Let's hope that a mod_python upgrade will help. Should I also upgrade to
> Python 2.4 as well? I develop with python 2.4 on my Ubuntu desktop, but
> deploy to Python 2.3.5 on the Debian server. 

I suspect changing to FileSession may solve some of your problems.

> Do you guys think I should use the newer version of python when building
> mod_python?

There are no python 2.4 specific features in mod_python 3.2.7, so from 
that perspective it's not necessary. We generally test with both python 
2.3.5 and 2.4.


BTW, how do you define high traffic? If you mentioned this before I 
missed it.

Jim