[mod_python] Background threads in mod_python

On 29/07/2006, at 8:08 AM, Christian Gross wrote:

> I have been using mod_python and am wondering about creating
> background threads. I can create the background thread, but if the
> thread requires a longer processing time than the request the
> background thread is exited.
>
> Is it possible to create a background thread and have it execute
> without being killed?

How do you know the thread is being killed? Are you sure it isn't the  
process
as a whole?

There isn't any fundamental reason why a thread can't outlive the  
life of the
request handler, but there are some common mistakes.

Main issue that you cannot pass the mod_python request object to such  
a thread
and have it hold a reference to it and use the request object beyond  
the life
of the request handler. This is because the internals of the request  
object
are destroyed when the request handler returns and an access to the  
request
object may result in the Apache child process then subsequently  
crashing.

Alternatively, if the Apache child process doesn't crash, you might  
at least
cause a Python exception to occur because the state of something in the
request object isn't as expected. This exception would effectively  
cause the
thread to exit unless the context in which the access was made to the  
request
object was protected by a try/except block.

In either case, if a Python exception occurs within the thread and it  
isn't
caught, it might not actually result in the details of the exception  
being logged.
This is because if the exception details are output, they go to  
standard error
and anything sent to standard error or standard output when running  
under
mod_python isn't guaranteed to show up anywhere. It may if flushing  
occurs,
appear in the Apache error log, but I wouldn't depend on it.  
Sometimes only
such output ends up in the Apache error log when Apache itself is  
shutdown
and those streams get flushed at that point.

In the above I talk about a Python exception occurring due to access  
to a
no longer valid request object, but even an uncaught exception in  
your code
will cause the same thing, with the thread appearing to exit but with  
nothing
necessarily being logged anywhere. What you want to do is to put a  
try/except
block around everything the thread does and if an exception occurs, then
use apache.log_error() to log something to the Apache error log  
indicating
that the exception occurs and the details of the exception.

One final thing, are you sure that the thread isn't deadlocking on  
something
and isn't just stalled? Have you actually used apache.log_error() to log
something at the exit point of the top level thread function to  
confirm that
it is really exiting? Note that exiting explicitly, or because of a  
Python
exception occurring within the thread is the only way the thread can  
truly
be stopped as there is no way in Python for a thread to be externally  
killed
by another thread anyway.

Graham