[mod_python] Session Hanging Problems

When you call req.write(), it defaults to flushing the data. Therefore
if you have a slow client or network issues then it may block waiting
to be able to flush out the data.

If your responses are not large, then use req.write(data, 0), ie.,
second argument of 0, to disable flushing. This will result in
response being buffered up in memory and only being flushed when the
whole request has completed.

Because it buffers in memory, you need to be careful with large
responses as can blow out memory usage.

Other alternative is to buffer all output yourself in StringIO and
then join it back together and send it all in one go when whole
response ready, and potentially after having released session locks.
This is actually more efficient, memory wise and potentially speed
wise, than allowing req.write() to buffer it because Apache output
filter buckets have more overhead than Python list for buffering.

Graham

On 04/12/2007, Harish Agarwal <harish at octopart.com> wrote:
>
> On Nov 26, 2007, at 1:38 PM, Graham Dumpleton wrote:
>
> > On 27/11/2007, Harish Agarwal <harish at octopart.com> wrote:
> >> I'm using session handling with ModPython 3.3.1.  Originally I was
> >> using DbmSession and have since transitioned to a custom MySQL
> >> Session
> >> handler.  With both session types, however, I've noticed that session
> >> initialization intermittently hangs (not forever, but takes as long
> >> as
> >> four minutes to complete), at a low-ish (a handful of times every
> >> hour, while receiving, say, on order of a thousand or so requests
> >> every hour) frequency which seems to scale with the amount of traffic
> >> we're receiving.  I had read that long DbmSession cleanups can cause
> >> problems, which is why I transitioned to the MySQL system, which
> >> takes
> >> < 1 second to complete, but I'm still noticing the long session init
> >> times.
> >>
> >>
> >> I put some debugging statements into the code and it seems to be
> >> related to session locking.  In particular, it is this function call
> >> in the lock method of the BaseSession class:
> >>
> >>
> >> _apache._global_lock(self._req.server, self._sid)
> >>
> >>
> >> which is taking some time to complete. I'm not familiar with
> >> _apache._global_lock (is it used to apply a mutex lock to the
> >> session?) and am having trouble finding information describing its
> >> usage - but it seems likely that this is the root cause.  In the past
> >> I've had problems with session locking but have since transitioned
> >> the
> >> code to ensure that only one session is created per request, as such:
> >>
> >> if not hasattr(req,'session'):
> >>             req.session = Session.MySQLSession(req)
> >>
> >>
> >> Can anyone tell me if this kind of behavior is normal or is
> >> indicative
> >> of some common configuration or coding error?  Any help would be
> >> greatly appreciated.
> >
> > Ignoring hangs, how long does your longest request normally take to
> > execute? Are you perhaps performing file uploads that take a
> > significant amount of time and are holding a session locked for the
> > whole period of the request?
> >
> > To at least allow some level of concurrency, from memory mod_python
> > holds a small pool of cross process mutex locks. Based on the session
> > ID (I think) it should consistently pick the same mutex lock each
> > time. Thus, if a request comes in with the same session ID it would be
> > blocked while the existing request for that session runs. If however
> > another request comes in with a different session ID, but where it
> > maps to the same mutex lock from the pool, it will also be blocked
> > until the request holding that lock has completed.
> >
> > What this means is that the number of mutex locks in the pool
> > effectively dictates how many parallel session based requests you can
> > have executed across the whole of the Apache server. Thus, if you have
> > requests that take a long time while still holding the session lock,
> > it can lock out other requests until it completes.
> >
> > One can makes things a bit better by increasing the number of mutex
> > locks in the pool, but you have to be careful not to create too many
> > in case it is using sysvsem and your OS doesn't allocate enough.
> >
> > The only other thing to do is to release the lock explicitly as soon
> > as you no longer need it and don't rely on the cleanup handler for the
> > request to unlock it.
> >
> > In other words, what your application is doing, your own code and how
> > you have written it could be the problem, and not necessarily
> > mod_python itself.
> >
> > If my memory has totally faded and my description is wrong, someone
> > please correct me. :-)
> >
> > Graham
>
>
> I put in some more debugging statements and I'm finding, very oddly,
> that the hanging is most likely due to a req.write taking quite a bit
> of time to complete.  I'm able to view the exact same webpage without
> experiencing a hang, so the hang seems to be fairly sporadic.
>
> Any ideas?
>
>
>
>
>