[mod_python] Persist database conn, or pers. variables - whereI put them ?

Graham Dumpleton grahamd at dscpl.com.au
Wed Sep 27 20:24:10 EDT 2006


Sean Davis wrote ..
> Richard Lewis wrote:
> > On Wednesday 27 September 2006 14:55, Sean Davis wrote:
> >   
> >> On Wednesday 27 September 2006 09:16, Fredrik Sandin wrote:
> >>     
> >>> I asked the same question some time ago:
> >>> http://modpython.org/pipermail/mod_python/2006-August/021924.html
> >>> Graham Dumpleton suggested to use XML-RPC.
> >>>
> >>> As far as I have understood mod_python, or actually Apache, was
> >>> not designed to allow for object persistency.
> >>>       
> >> Is this actually true (at least the Apache part)?  Under mod_perl (I'm
> a
> >> mod_python total newbie), object persistence can be accomplished in
> several
> >> ways including global variables that remain in shared memory (but this
> is
> >> only within a single child), a database, in an in-memory or on-disk
> cache,
> >> or in a session (just an interface to a cache or database).  Depends
> on
> >> your definition, but Apache, it seems to me, is absolutely suited to
> allow
> >> object persistence.
> >>
> >>     
> > Persistance is all right: you can use global variables as you suggested.
> >
> > The problem comes at tidying up time. Apache provides no method of performing
> > operations when the server shuts down. This is especially relevant with
> > things like open database connections 
> Richard,
> 
> Thanks for the clarification. 
> 
> Under mod_perl, there is Apache::DBI that overrides the connect() method
> of the DBI module to allow reuse of connections (maintaining a 
> persistent connection).  When the server process dies, the DESTROY() 
> method is called and the connections are cleaned up.  Would this be 
> treated differently under mod_python for some reason, if a database 
> connection object had a destroy() method that included a disconnect?  I
> have to admit that I don't understand many of the intricacies, but it 
> seems that if it can be done in perl under mod_perl, there is likely a
> way to do it under python and mod_python.

Whether one can reliably cause actions to be run on server shutdown in
mod_python is a contentious issue as the moment. I have been a bit lax in going
back and doing some further testing on the issue to settle it one way or the
other and no one else has stepped up to the plate to do the required analysis.

What I can say is the following.

1. Even where a mechanism is available for triggering actions to be called on
shutdown, there is no guarantee that the actions will be called purely because
the Apache parent process can make a decision that a child process isn't
shutting down properly by itself and it will send it a KILL signal to cause it
to immediately quit. Processes can always crash for no good reason as well.

Thus, one shouldn't write code that absolutely requires such clean shutdowns to
occur. Any startup code should always be written to deal with a prior unclean
shutdown and do any special cleanup first if required.

Also, sometimes a persons belief that cleanup code is required has been
based on a mistaken belief that if not done then resources would be leaked.
Certainly in the case of database connections, this shouldn't be the case as
the server side will detect that client connections have gone away and will
automatically cleanup on the server side even if a clean close of the connection
hadn't been done.

2. In mod_python 3.3, any Python interpreter instances will not be destroyed on
child process shutdown. The interpreter instances are being destroyed in
mod_python prior to 3.3, but doing so can cause the Apache child process to
hang, as this can get triggered from within a signal handler in certain
circumstances and if there is an active request handler for mod_python running
at the time. the signal handler will block due to not being able to acquire the
Python GIL. Thus, calling Py_Finalize() for interpreters is no longer being
done at this point.

The implications of this are that Python objects are not being destroyed on
child process shutdown, thus you cannot rely on the destructor of Python
objects doing anything to cleanup resources.

3. In mod_python, there are the functions apache.register_cleanup() and
req.server.register_cleanup() for registering actions to be done on server
cleanup. However, the trigger for these being called is the destruction of
the childs Apache memory pool. As with 2, this seems in some cases to be
triggered from a signal handler and thus can similarly result in the Apache
child process hanging on shutdown.

See:

  http://issues.apache.org/jira/browse/MODPYTHON-109

for more details.

Thus, the question has been whether if no guarantees can be given as to
whether cleanup actions will even be called and if an attempt is made to
call them, it still may result in them not being run because of the Apache
child process hanging due to locking problems, should the feature even
be provided.

Now it is possible that prior analysis of the Apache source code and how
these cleanup actions are triggered could be wrong, or based on old Apache
code where it worked a bit differently. As I said though, no one else has
stepped up to do a proper analysis of the Apache code and do testing in
respect of the issue and I haven't had a chance lately to look at it again.
The other core developers have also been quite busy of late. Without this
further analysis, we will not know if 2 and 3 are really problems or not.

Graham



More information about the Mod_python mailing list