[mod_python] forks, daemons and other beasts

Sorry everyone, a long post follows :)


Many times there is need for a mod_python program to respond (send a
page) to the client after receiving a request and at the same time
start a subprocess which typically takes quite long and waiting for it
to finish is not necessary. So the situation is this: (1) request
comes from client (2) server processes the request (3) server starts a
subprocess and don't wait for it to finish (4) send a response to the
client. Actually, (3) and (4) could be interchanged.

What is the best way to do this?

One could of course use the exec* and spawn* family of functions, but
in that case one would need to have a separate program to run and all
variables (lists, classes, etc.) used by the program should be
translated to command line arguments which can be cumbersome.

I've learned from a recipe in the python cookbook,
http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/66012, that
this sort of thing in a python application would be done by forking
the process twice (just like in unix) like this:


import sys, os

def daemonize ( stdin='/dev/null', stdout='/dev/null', stderr='/dev/null' ):

    # Perform first fork.
    try:
        pid = os.fork()
        if pid > 0:
            sys.exit(0) # Exit first parent.
    except OSError, e:
        sys.stderr.write("fork #1 failed: (%d) %s\n" % (e.errno, e.strerror))
        sys.exit(1)

    # Decouple from parent environment.
    os.chdir("/")
    os.umask(0)
    os.setsid()

    # Perform second fork.
    try:
        pid = os.fork()
        if pid > 0:
            sys.exit(0) # Exit second parent.
    except OSError, e:
        sys.stderr.write("fork #2 failed: (%d) %s\n" % (e.errno, e.strerror))
        sys.exit(1)

    # The process is now daemonized, redirect standard file descriptors.
    for f in sys.stdout, sys.stderr: f.flush()
    si = file(stdin, 'r')
    so = file(stdout, 'a+')
    se = file(stderr, 'a+', 0)
    os.dup2(si.fileno(), sys.stdin.fileno())
    os.dup2(so.fileno(), sys.stdout.fileno())
    os.dup2(se.fileno(), sys.stderr.fileno())

def this_will_run_as_a_daemon( ):
    # some stuff here


daemonize( '/dev/null', '/tmp/daemon.log', '/tmp/daemon.log' )
this_will_run_as_a_daemon( )


All of this is working fine with python, the result is what one would
expect, whatever is in this_will_run_as_a_daemon( ) will run as a
daemon in the background.

Now I have difficulty with doing a similar thing with mod_python and
the publisher handler. I tried the following (the above daemonize
function is in daemon.py):

from daemon import *

def foo( req ):
    daemonize( '/dev/null', '/tmp/daemon.log', '/tmp/daemon.log' )

    # some stuff here that should run in the background
    # and the program should just start it and carry on

    return "okay"

and then accessing it by a client gives strange behaviour, either does
it what it supposed to do or gives the following error:


Mod_python error: "PythonHandler mod_python.publisher"

Traceback (most recent call last):

  File "/usr/lib/python2.3/site-packages/mod_python/apache.py", line
299, in HandlerDispatch
    result = object(req)

  File "/usr/lib/python2.3/site-packages/mod_python/publisher.py",
line 136, in handler
    result = util.apply_fs_data(object, req.form, req=req)

  File "/usr/lib/python2.3/site-packages/mod_python/util.py", line
361, in apply_fs_data
    return object(**args)

  File "/var/www/html/python/test.py", line 5, in d
    daemonize('/dev/null', '/tmp/moddaemon.log', '/tmp/moddaemon.log')

  File "/var/www/html/python/daemon.py", line 27, in daemonize
    sys.exit(0) # Exit second parent.

SystemExit: 0

I guess part of the problem is that anything after daemonize( ) really
gets 'daemonized' including the return "okay" statement. But then how
can it work once in a while?

Anyway, I have the feeling that there must be a more sane and probably
safe way of achieving what I would like by a much simpler method.

Any ideas?