[mod_python] Re: forks, daemons and other beasts

Mike Looijmans nlv11281 at natlab.research.philips.com
Mon Feb 13 03:01:44 EST 2006

>>Run a small separate daemon process which embeds an  XML-RPC
>>server and have your web pages use XML-RPC to call that daemon
>>process to trigger off the task. The daemon process can still fork exec
>>as necessary.
>>This way you eliminate a fork in Apache child process which may be
>>problematic because of the socket connections etc that are inherited.
> Well, that still looks like to me that I'm shooting a bird with a
> cannon. The reason is that I need to do something really simple: users
> upload a zip file (remember my questions about the zipfile module? :))
> and should get an acknowledgement that the upload was successful. The
> server should start unpacking the archive, check if the files are okay
> and move them to a directory. All of this can take quite long, so the
> acknowledgement should be sent before this unpacking/checking/moving
> finishes.

(Uh... Just one observation: They UPLOAD a ZIP file, and you expect the 
unpacking to take longer than that?)

You might consider using the latest mod_python 3.2.7 modification that 
supports file "streaming". You can unpack the ZIP while it's being 
uploaded. This also prevents using TEMP space. And when the upload is 
ready, the files are there as well. I tested this method for uploading 
multi-gigabyte TAR files.

The trick is to add a file callback to the FieldStorage object. I have 
attached a sample upload PSP script that just counts bytes. I'll leave 
it up to your imagination to feed the data into a ZipFile object.

Attempting to fork extra threads or processes from within an Apache 
instance will bring you lots of troubles - and you never know what 
happens in future versions.

Mike Looijmans
-------------- next part --------------
import fmt
  <%= fmt.style %>
<p><a href="upload.psp?reset=log">Reset</a></p>
<form enctype="multipart/form-data" method="POST" action="upload.psp?doit=yes">
  <input type=file size=64 name="archivefile"><br>
  <input type=file size=64 name="archivefile"><br>
  <input type=file size=64 name="archivefile">
  <input type=submit name=action value="Create">


from mod_python import util

class FileCounter:
    def __init__(self, name):
        self.size = 0
        self.blocks = 0
        self.blocksize = 0
        self.name = name
    def write(self, data):
        l = len(data)
        self.size += l
        self.blocks += 1
        if l > self.blocksize: self.blocksize = l
    def seek(self, pos):

def make_file(filename):
    req.write('<p>makefile: ' + str(filename) + '</p>\n')
    return FileCounter(filename)

frm = util.FieldStorage(req, file_callback=make_file)
for afile in frm.getlist('archivefile'):
    req.write('<p>File name: <b>%s</b>\n' % afile.filename)
    req.write('read: <b>%d</b> bytes, <b>%d</b> blocks max blocksize: <b>%d</b></p>\n' % (afile.file.size, afile.file.blocks, afile.file.blocksize))

# end if


More information about the Mod_python mailing list