[mod_python] Due to design of code util.FieldStorage causes MemoryError on V.Large files

Jorey Bump list at joreybump.com
Fri Mar 25 12:08:01 EST 2005


Barry Pearce wrote:

> Ive just tried to upload an .iso using FieldStorage - an unfortunately 
> this causes some problems - namely it just calls 'readline' without a 
> specific sizing.
> 
> Im going to take a look at the code and try to improve it so that it 
> would cope with multi-GB files...

> Any thoughts before I plough headlong into this?

I would probably never allow transfers of this type using HTTP, so be 
sure to provide sane defaults and a limiting mechanism.

Also, considering the amount of time it would take to complete the 
transfer on just about any typical connection, how many simultaneous 
transfers would be possible in a default apache setup before it 
effectively stops serving pages?

How will you handle uploads of the same file when a luser keeps 
submitting the same form? For small files, this is a nuisance, but for 
large ones it could be a catastrophe.

Finally, consider every aspect of the worst case scenario and don't 
discount human stupidity. For example, a user once tried to send me an 
email with a 40GB attachment (the entire C: drive) using Outlook 
Express. His mail server properly identified the maximum size it would 
handle, but OE stubbornly continued to transfer bytes and attempted to 
clear out its Outbox all night long (he had gone to bed). In the 
morning, his provider gave him an angry call. We took the machine 
offline and spent a couple of hours trying to purge the message from OE 
(harder than you would think). Test your code with every browser 
available to you, and do every stupid thing you can think of before 
giving ordinary users the power to DoS your system.





More information about the Mod_python mailing list