[mod_python] Re: forks, daemons and other beasts

Mike Looijmans
Philips Natlab / Topic Automation


Daniel Nogradi wrote:
> The unpacking itself doesn't take that long, but until the unpacked files reach their final destination (directories, databases, etc) will take quite some time as they have to be processed in some way. But I agree, uploading takes much longer, but I would like to make the waiting time for the users as short as possible, and in principle the unpacking/processing/etc can be done after they receive an acknowledgement page.
> 

And like all users in the world, they'll go to some status page and hit 
the "refresh" button 5 times a second to see if it's there yet.

>>You might consider using the latest mod_python 3.2.7 modification that 
>>supports file "streaming". You can unpack the ZIP while it's being 
>>uploaded. This also prevents using TEMP space. And when the upload is 
>>ready, the files are there as well. I tested this method for uploading 
>>multi-gigabyte TAR files.
...
> Thanks a lot for the code, I didn't even think about such an approach, this would be the best of course.

It would probably also be the fastest method performance wise (with my 
big tar files, untar-as-you-go was about 2x to 3x faster than upload and 
untar in separate steps) in the end, your users will be waiting much 
shorter, and once they get the acknowledgement page, the work has really 
been done.

Forking off processes from a HTTP deamon just does not sound "right" to 
me. It will also prevent your program from ever working on a non-unix 
platform. But that's more a religious thing I guess...