[mod_python] Due to design of code util.FieldStorage causes MemoryError on V.Large files

Jim Gallacher jpg at jgassociates.ca
Thu Nov 3 17:22:01 EST 2005


Fixed in 3.2. See http://issues.apache.org/jira/browse/MODPYTHON-40

Jim

Alexis Marrero wrote:
> Just to through some information out there...
> 
> This is the function that reads the "uploaded" file and writes to disk.
> 
>     def read_to_boundary(self, req, boundary, file):
>         delim = ""
>         line = req.readline()
>         sline = line.strip()
>         last_bound = boundary + "--"
>         while line and sline != boundary and sline != last_bound:
>             odelim = delim
>             if line[-2:] == "\r\n":
>                 delim = "\r\n"
>                 line = line[:-2]
>             elif line[-1:] == "\n":
>                 delim = "\n"
>                 line = line[:-1]
>             file.write(odelim + line)
>             line = req.readline()
>             sline = line.strip()
> 
> What about if there are no "\n" delimiters in the file and the file  is 
> 512MB?
> 
> I know that someone already suggested instead of readline() to do  sized 
> read(MAX_SIZE) ... Is this already implemented in mod_python 3.2?
> 
> 
> /amn
> 
> On Nov 3, 2005, at 11:34 AM, Alexis Marrero wrote:
> 
>> Barry,
>>
>> I'm also having a MemoryError on line:
>>
>>   File "/Library/Frameworks/Python.framework/Versions/2.4/lib/ 
>> python2.4/site-packages/mod_python/util.py", line 175, in __init__
>>     self.read_to_boundary(req, boundary, file)
>>
>>   File "/Library/Frameworks/Python.framework/Versions/2.4/lib/ 
>> python2.4/site-packages/mod_python/util.py", line 206, in  
>> read_to_boundary
>>     sline = line.strip()
>>
>> MemoryError
>>
>>
>> The file that is uploaded is 550MB.
>>
>> BTW, the Content-Disposition header has a filename attribute.
>>
>> What is the status with the changes that you mention on doing?
>>
>> My web application must be able to support attachments of arbitrary  
>> size.
>>
>> Any comments?
>>
>> /amn
>> _______________________________________________
>> Mod_python mailing list
>> Mod_python at modpython.org
>> http://mailman.modpython.org/mailman/listinfo/mod_python
>>
> 
> _______________________________________________
> Mod_python mailing list
> Mod_python at modpython.org
> http://mailman.modpython.org/mailman/listinfo/mod_python
> 



More information about the Mod_python mailing list