[mod_python] Session Hanging Problems

Harish Agarwal wrote:
> 
> On Nov 26, 2007, at 1:38 PM, Graham Dumpleton wrote:
> 
>> On 27/11/2007, Harish Agarwal <harish at octopart.com> wrote:
>>> I'm using session handling with ModPython 3.3.1.  Originally I was
>>> using DbmSession and have since transitioned to a custom MySQL Session
>>> handler.  With both session types, however, I've noticed that session
>>> initialization intermittently hangs (not forever, but takes as long as
>>> four minutes to complete), at a low-ish (a handful of times every
>>> hour, while receiving, say, on order of a thousand or so requests
>>> every hour) frequency which seems to scale with the amount of traffic
>>> we're receiving.  I had read that long DbmSession cleanups can cause
>>> problems, which is why I transitioned to the MySQL system, which takes
>>> < 1 second to complete, but I'm still noticing the long session init
>>> times.
>>>
>>>
>>> I put some debugging statements into the code and it seems to be
>>> related to session locking.  In particular, it is this function call
>>> in the lock method of the BaseSession class:
>>>
>>>
>>> _apache._global_lock(self._req.server, self._sid)
>>>
>>>
>>> which is taking some time to complete. I'm not familiar with
>>> _apache._global_lock (is it used to apply a mutex lock to the
>>> session?) and am having trouble finding information describing its
>>> usage - but it seems likely that this is the root cause.  In the past
>>> I've had problems with session locking but have since transitioned the
>>> code to ensure that only one session is created per request, as such:
>>>
>>> if not hasattr(req,'session'):
>>>             req.session = Session.MySQLSession(req)
>>>
>>>
>>> Can anyone tell me if this kind of behavior is normal or is indicative
>>> of some common configuration or coding error?  Any help would be
>>> greatly appreciated.
>>
>> Ignoring hangs, how long does your longest request normally take to
>> execute? Are you perhaps performing file uploads that take a
>> significant amount of time and are holding a session locked for the
>> whole period of the request?
>>
>> To at least allow some level of concurrency, from memory mod_python
>> holds a small pool of cross process mutex locks. Based on the session
>> ID (I think) it should consistently pick the same mutex lock each
>> time. Thus, if a request comes in with the same session ID it would be
>> blocked while the existing request for that session runs. If however
>> another request comes in with a different session ID, but where it
>> maps to the same mutex lock from the pool, it will also be blocked
>> until the request holding that lock has completed.

That is an accurate summary. The mutex index is calculated by the 
following pseudo code:

index = (hash(session_id) % (nlocks - 1) + 1)

nlocks is 8 by default. Mutex index 0 is reserved for dbm file locking 
so you really only have 7 mutexes available.

The number of locks can be set with the "PythonOption 
mod_python.mutex_locks" config directive or at compile time.

>> What this means is that the number of mutex locks in the pool
>> effectively dictates how many parallel session based requests you can
>> have executed across the whole of the Apache server. Thus, if you have
>> requests that take a long time while still holding the session lock,
>> it can lock out other requests until it completes.
>>
>> One can makes things a bit better by increasing the number of mutex
>> locks in the pool, but you have to be careful not to create too many
>> in case it is using sysvsem and your OS doesn't allocate enough.
>>
>> The only other thing to do is to release the lock explicitly as soon
>> as you no longer need it and don't rely on the cleanup handler for the
>> request to unlock it.
>>
>> In other words, what your application is doing, your own code and how
>> you have written it could be the problem, and not necessarily
>> mod_python itself.
>>
>> If my memory has totally faded and my description is wrong, someone
>> please correct me. :-)

It's all good. :)

>> Graham