[mod_python] Due to design of code util.FieldStorage causesMemoryError on V.Large files

Alexis Marrero wrote:
> This is the function that reads the "uploaded" file and 
> writes to disk.
> 
>      def read_to_boundary(self, req, boundary, file):
>          delim = ""
>          line = req.readline()
>          sline = line.strip()
>          last_bound = boundary + "--"
>          while line and sline != boundary and sline != last_bound:
>              odelim = delim
>              if line[-2:] == "\r\n":
>                  delim = "\r\n"
>                  line = line[:-2]
>              elif line[-1:] == "\n":
>                  delim = "\n"
>                  line = line[:-1]
>              file.write(odelim + line)
>              line = req.readline()
>              sline = line.strip()
> 
> What about if there are no "\n" delimiters in the file and the file  
> is 512MB?
> 
> I know that someone already suggested instead of readline() to do  
> sized read(MAX_SIZE) ... Is this already implemented in 
> mod_python 3.2?

I'm not sure if it's been implemented in 3.2 or not, but just to prove
you're not imagining things, this is a known bug in Python's cgi module,
as well:
http://sourceforge.net/tracker/?func=detail&aid=1112549&group_id=5470&at
id=105470. See CherryPy's _cpcgifs module for a sample fix:
http://www.cherrypy.org/file/trunk/cherrypy/_cpcgifs.py. The same
changes (provide a sizehint to the readline call) could probably be done
to mod_python.util. 


Robert Brewer
System Architect
Amor Ministries
fumanchu at amor.org