[mod_python] Session class or Memcache to reduce load?

Thanks everybody for responding.
I implemented 
> For starters, you should use req.sendfile instead of manually reading
> / sending the file in Python.

This reduced load average from 13.0 to 10.0

> Using DbmSession may actually increase your load for a couple of
> reasons. For starters access to the database is serialized by a global
> lock which may be a bottleneck.

In this regard, I have another question: should each request open it's own
MySQL connection (the connection to our DB slave servers), or only one
connection should be opened, and passed around between requests using
DbmSession or some other means?

> -----Original Message-----
> From: Jim Gallacher [mailto:jpg at jgassociates.ca]
> Sent: Wednesday, August 08, 2007 7:55 AM
> To: Alec Matusis
> Cc: mod_python at modpython.org
> Subject: Re: [mod_python] Session class or Memcache to reduce load?
> 
> Other folks have already replied with good advice, but I want to add a
> few comments on session handling.
> 
> Alec Matusis wrote:
> > Hello, I am sorry if my question is too basic. I would like to reduce
> the
> > load on apache 2.0 (on 2.6.9 linux) that is running with prefork MPM.
> > There are two main things that are causing the load:
> >
> > a) Thumbnail images that are requested repeatedly
> > b) A simple DB query is necessary to locate an image file after the
> request.
> > The result of the query does not change. DB is located on another
> machine.
> >
> > My first question, do we need to cache \thumbnail images at all, or
> the
> > file-caching by the OS is sufficient?
> >
> > Second question, to cache the results of the query, should we use
> > mod_python's Session class ( wich will use DbmSession since we are
> using
> > prefork MPM), or memcache?
> 
> Using DbmSession may actually increase your load for a couple of
> reasons. For starters access to the database is serialized by a global
> lock which may be a bottleneck.
> 
> More seriously, a cleanup of expired sessions is performed after
> approximately 1000 requests per child. During this cleanup the db will
> also be locked. If you have a really large number of  sessions (greater
> than 10,000 for example) the time required for the cleanup will not be
> trivial and the rate at which requests are served may drop
> significantly. I did some benchmarking on this and saw the rate at
> which
> requests were served drop by 90% in some circumstances (IIRC). If
> you're
> really interested you can look in the mail archives on the developer's
> list.
> 
> This cleanup side effect was something I noticed when developing the
> FileSession class (introduced in 3.2.7). FileSession has a couple of
> features that avoid the cleanup problems. DbmSession can't be fixed due
> to some limitations of dbm.
> 
> If you decide you do need to use sessions make sure you run some
> benchmarks first. If you see a problem then either use FileSession or
> create a backend for a proper relational database where the cleanup can
> be extremely fast.
> 
> Jim