[mod_python] Session class or Memcache to reduce load?

> Are you currently using DbmSession? 

No, we don't.

> How many concurrent requests? 

This is the output of /server-status (not at the peak load. At peak it can
be up to 120-130)

96.3 requests/sec - 363.1 kB/second - 3862 B/request 
60 requests currently being processed, 21 idle workers 

C.RW.CW__.C_RCWCCC__CCCWCWC_CCW_CC_CCC_C_C.CCC__WCRCCC._CCR..WW_
.__WCCC.CCC.CC....R....._.W......C..............._....C.........
....C....RC...._._..........._W.................................
................................................................

This is what we have in httpd.conf:

<IfModule prefork.c>
StartServers          10
MinSpareServers       5
MaxSpareServers       10
MaxClients            180
MaxRequestsPerChild   10000
MaxMemFree      2048
</IfModule>


> What version of mod_python you are using? 

Server Version: Apache/2.0.54 (Unix) mod_python/3.1.4 Python/2.4.1
We are planning an upgrade to Apache 2.2 and the latest mod_python very
soon.

> Let me know and I'll pass along
> some ideas for optimizing session handling.

Thank you.

> -----Original Message-----
> From: Jim Gallacher [mailto:jpg at jgassociates.ca]
> Sent: Wednesday, August 08, 2007 3:37 PM
> To: Alec Matusis
> Cc: mod_python at modpython.org
> Subject: Re: [mod_python] Session class or Memcache to reduce load?
> 
> Alec Matusis wrote:
> > Thanks everybody for responding.
> > I implemented
> >> For starters, you should use req.sendfile instead of manually
> reading
> >> / sending the file in Python.
> >
> > This reduced load average from 13.0 to 10.0
> 
> I can see why you might want to speed things up! :)
> 
> >> Using DbmSession may actually increase your load for a couple of
> >> reasons. For starters access to the database is serialized by a
> global
> >> lock which may be a bottleneck.
> >
> > In this regard, I have another question: should each request open
> it's own
> > MySQL connection (the connection to our DB slave servers), or only
> one
> > connection should be opened, and passed around between requests using
> > DbmSession or some other means?
> 
> DbmSession won't get you anywhere here as you can't pickle the
> connection object. Even if you could the connection would still need to
> be re-established when the pickle is loaded so you are no further
> ahead.
> Furthermore the overhead of reading + un-pickling is likely greater
> than
> just opening and closing your DB connection anyway, so unless you are
> already using session handling you could end up with lower performance.
> Establishing a MySQL connection is pretty fast although this may not be
> the case for other databases.
> 
> Connection pooling or caching has been discussed on the list in the
> past. You might want to dig through the mail archives. Just be aware
> that when using the prefork-mpm each child will get it's own MySQL
> connection. If you are under heavy load and alot of children get
> spawned
> you could end up hitting a MySQL resource limit for the number of
> connections. This would be bad. ;)
> 
> Connection pooling would be more effective with the worker or threaded
> mpms, where the number of children are limited. Threads in the same
> child process can draw from the same pool so it's possible to actually
> manage the pool.
> 
> Are you currently using DbmSession? How many concurrent requests? What
> version of mod_python you are using? Let me know and I'll pass along
> some ideas for optimizing session handling.
> 
> Jim