[mod_python] Session class or Memcache to reduce load?

Other folks have already replied with good advice, but I want to add a 
few comments on session handling.

Alec Matusis wrote:
> Hello, I am sorry if my question is too basic. I would like to reduce the
> load on apache 2.0 (on 2.6.9 linux) that is running with prefork MPM.
> There are two main things that are causing the load:
> 
> a) Thumbnail images that are requested repeatedly 
> b) A simple DB query is necessary to locate an image file after the request.
> The result of the query does not change. DB is located on another machine.
> 
> My first question, do we need to cache \thumbnail images at all, or the
> file-caching by the OS is sufficient?
> 
> Second question, to cache the results of the query, should we use
> mod_python's Session class ( wich will use DbmSession since we are using
> prefork MPM), or memcache?

Using DbmSession may actually increase your load for a couple of 
reasons. For starters access to the database is serialized by a global 
lock which may be a bottleneck.

More seriously, a cleanup of expired sessions is performed after 
approximately 1000 requests per child. During this cleanup the db will 
also be locked. If you have a really large number of  sessions (greater 
than 10,000 for example) the time required for the cleanup will not be 
trivial and the rate at which requests are served may drop 
significantly. I did some benchmarking on this and saw the rate at which 
requests were served drop by 90% in some circumstances (IIRC). If you're 
really interested you can look in the mail archives on the developer's list.

This cleanup side effect was something I noticed when developing the 
FileSession class (introduced in 3.2.7). FileSession has a couple of 
features that avoid the cleanup problems. DbmSession can't be fixed due 
to some limitations of dbm.

If you decide you do need to use sessions make sure you run some 
benchmarks first. If you see a problem then either use FileSession or 
create a backend for a proper relational database where the cleanup can 
be extremely fast.

Jim