[mod_python] Spawning Long Running Processes

On 04/06/2005, at 4:56 AM, Michael Rushton wrote:

> Hello,
> What is the best way to spawn a process from a mod_python that could 
> take several hours to complete?

How much data needs to be communicated between mod_python running under
Apache and the process doing the processing and generation of the file
at initiation of the process and then later?

> I am trying to write a handler that when accessed for the first time 
> spawns a long running process that writes to a file. On subsequent 
> accesses, the handler would show the contents of the file produced by 
> the long process, refreshing the page occasionally with meta refresh 
> tags.
>
> I have had a stab at achieving this by double forking the process. 
> This however, has had the disadvantage that for really long running 
> processes   the forked processes still seem to be bound to port 80 
> thus preventing httpd daemon restarts. A problem when you want to 
> update apache configuration settings.
>
> What am I doing wrong? And what is the accepted way for spawning long 
> lived processes from mod_python?
>
> I would be very grateful if someone could point me to some source 
> examples that show how to achieve this.

I would avoid forking and perhaps look at a system where you always
have a persistent back end process running all the time. Ie., it would
not be created just at the time the request came in but would already
exist, ready and waiting.

When a request comes in, the mod_python code could trigger an XML-RPC
request, or some other form of interprocess communications, which asks
the long lived back end process to start off what it needs to do to 
create
the file. Provided that the back end process is not written so as to
completely block while generating the file, a subsequent request could
again use something like XML-RPC to query whether it has finished or
at least where it is up to. When it gets a positive response that the
back end processing has finished, if it is a small amount of data it
could be returned in the XML-RPC response, or if a lot, the mod_python
side could just be returned the name of the file which it then tells
Apache to deliver back as the response.

For one example of a package that could be used to implement such a
back end system, have a look at OSE (http://ose.sourceforge.net). This
has the means using event driven models of programming or appropriate
use of threads to allow a long running job to run in the back end system
but still allow the system to be communicable using XML-RPC or over
its own messaging system. Both these communication means can be accessed
from mod_python handlers.

Anyway, have a look at the documentation for the Python wrappers for
OSE. You may not like OSE because at its core it is C++ (not that you
have to even use C++), but it may give you some ideas. Can help if you
are interested in going this path.

Graham