[mod_python] Session class or Memcache to reduce load?

Alec Matusis wrote:
> Thanks everybody for responding.
> I implemented 
>> For starters, you should use req.sendfile instead of manually reading
>> / sending the file in Python.
> 
> This reduced load average from 13.0 to 10.0

I can see why you might want to speed things up! :)

>> Using DbmSession may actually increase your load for a couple of
>> reasons. For starters access to the database is serialized by a global
>> lock which may be a bottleneck.
> 
> In this regard, I have another question: should each request open it's own
> MySQL connection (the connection to our DB slave servers), or only one
> connection should be opened, and passed around between requests using
> DbmSession or some other means?

DbmSession won't get you anywhere here as you can't pickle the 
connection object. Even if you could the connection would still need to 
be re-established when the pickle is loaded so you are no further ahead. 
Furthermore the overhead of reading + un-pickling is likely greater than 
just opening and closing your DB connection anyway, so unless you are 
already using session handling you could end up with lower performance. 
Establishing a MySQL connection is pretty fast although this may not be 
the case for other databases.

Connection pooling or caching has been discussed on the list in the 
past. You might want to dig through the mail archives. Just be aware 
that when using the prefork-mpm each child will get it's own MySQL 
connection. If you are under heavy load and alot of children get spawned 
you could end up hitting a MySQL resource limit for the number of 
connections. This would be bad. ;)

Connection pooling would be more effective with the worker or threaded 
mpms, where the number of children are limited. Threads in the same 
child process can draw from the same pool so it's possible to actually 
manage the pool.

Are you currently using DbmSession? How many concurrent requests? What 
version of mod_python you are using? Let me know and I'll pass along 
some ideas for optimizing session handling.

Jim