[mod_python] Memory error

export at hope.cz wrote ..
> I spent some time to write an input filter hoping  that an input filter
> will
> help me upload large files without memory errors.
> But it does NOT help.
> When I checked Apache error log today I saw:
> MemoryError, referer: http://127.0.0.1/mod/mptest.py?17644
> 
> ( mptest.py is handler I use for uploading .)
> 
> 
>  I use
> Apache 2.2.4
> Mod_python 3.3.0.b
> Python 2.3.5
> 
> 
> Input filter looks like this
> 
> def inputfilter(filter):
> 	if filter.req.method != 'POST':
> 		filter.pass_on()
> 		return
> 	f=open('uploadedfile','ab')

Where the file will be placed is unpredictable. You should use an
absolute path to the file. You should also be using some unique
name for the actual file else concurrent uploads will interfere with
each other. The name you use for the file should be passed through
to your request handler by setting an attribute in filter.req, ie.,

  filter.req.uploadfilename = "someuniquename"
 	
> 	s = filter.read()
> 	while s:
> 		f.write(s)
> 		f.flush()
> 		filter.write(s)
> 		filter.flush()

If you are wanting the data to be written to a file but not take up
memory in Apache as well, you shouldn't be also writing the data
onto the filter object as by doing so it isn't being discarded and
your process will run out of memory.

> 		s = filter.read()
> 	if s is None:
> 		filter.close()
> 		f.close()
> 
> Can anyone help? Will input filter really help or not?
> Does anyone have any experience with uploading large files( 100 MB and
> higher)?

I can't remember if it was you again or someone else has been asking about
input filters and large file uploads, but I have already pointed out Tramline
in a post recently, which can be used as a working example of how to do this
sort of stuff. The only difference in Tramline was that the final request was
being proxied through to a backend, so it was setting the filename in a header
rather than an attribute of the request object. The URL for Tramline again is:

  http://www.infrae.com/products/tramline

Graham