[mod_python] Input filter

Graham Dumpleton grahamd at dscpl.com.au
Tue Dec 26 17:09:29 EST 2006

On 26/12/2006, at 11:40 PM, export at hope.cz wrote:

> Graham,
> Thank you for your reply.
> I studied the example  at http://www.modpython.org/pipermail/ 
> mod_python/2006- April/020870.html
> Now I understand a little more.
> There an author uses
> streambuffer = filter.req.streambuffer
> So, I think that streambuffer is a method of request(req) object.
> Am I right?


An important thing to know about filters is that the filter function can
be called more than once for the same request. That is, it processes  
in chunks.

Also, for each of those calls of the filter function, although the  
'filter' object
is a distinct object each time, the 'filter.req' object is the same  
for all requests.
As a consequence, it is possible to cache data within the  
'filter.req' object
so that it is available across invocations of the filter functions.

With that in mind, look at the code again and you will see that  
within the
'try' block the filter function tries to access  
'filter.req.streambuffer'. That will
fail on the first call of the filter function for the request as that  
attribute will
not exist. As a result, the 'except' block is run which initialises the
attribute with an instance of a StringIO object.

Having ensure that the attribute exists, on each call to the filter  
the chunks of data are added to the StringIO object instance until no  
data is found. At that time the data in the StringIO object is turned  
into one string and processed.

In output filter examples all processing of the data is done in  
memory. This
isn't want you want, so you would instead substitute  
with an open file handle and thus write each chunk of data out to  
disk and
close the handle when done.

Writing out to disk was what Tramline was doing, with it then only  
through a header to subsequent request handler containing a value to
identify the file on disk.

> Is there a description of  all request (req) methods.I could not  
> find it in mod_python doc.

Those you need to know about are:


If you can't find it in there, it probably isn't actually an  
attribute or method.

> Is there any advantage to use
> filter.req.streambuffer
> instead of
> filter.read()  ?

Read over description above and check code again. The 'filter.read()'  
is still used, with data being accumulated in  
'filter.req.streambuffer' as

> Yet, after studing the filter examples,
> I do not understand how to verify that input data( from POST  
> request)  is written directly to disk and not loaded to RAM first  
> and only after  the whole file is uploaded then write to disk
> Can you please explain?

You really need to dig through the code examples and understand how
filters work. If you can work that out and see where the data is  
going you
should be able to work that out yourself. Obviously the output filter  
you are using aren't even opening files on disk to write to so  
nothing can
be getting to disk in those.

BTW, did you investigate the other option I pointed out in the  
response to
your first email some time back. Ie., the file callbacks in  
FieldStorage class?
Or are you trying to use mod_python.publisher and was that not an

The examples how to use FieldStorage so as to intercept uploaded files
and have them sent direct to disk can be found at:


BTW, are you the same person as 'Lad' posting on comp.lang.python
asking about this? I don't want to be answering in two different  
places if
you are the same person.


> Thank you
> La.
> >
> > I specifically said in my email:
> >
> >    Just because Tramline is intended more for cases where Apache is
> >    front ending a backend application doesn't mean it isn't a valid
> > example
> >    of input filters, you would just need to adapt the ideas to your
> > needs.
> >
> > So I already said that it isn't exactly what you require. What you
> > should do
> > is read through the code and the mod_python documentation and learn
> > from it so you can work out how you can customise it to do what you
> > want.
> > You do not need to be using mod_proxy to use input filters,  
> Tramline was
> > using it for specific reasons of its own.
> >
> > The relevant part of the mod_python documentation is:
> >
> >   http://www.modpython.org/live/current/doc-html/pyapi-filter.html
> >
> > The example is for an output filter, but input filters work almost
> > the same.
> >
> > Also look back through past mailing lists post by using search  
> box on
> > the
> > mod_python web site. A slightly more complicated example of an  
> output
> > filter is:
> >
> >    http://www.modpython.org/pipermail/mod_python/2006-April/ 
> 020870.html
> >
> > Overall though, the Tramline code is the most complicated example  
> of an
> > input filter I have seen.
> >
> > Graham
> >
> > >>> Can anyone give me an example of an Input filter?
> > >>> I would like to check if a  file being uploaded is large and  
> if so,
> > >>> I would like to write the file
> > >>> directly on hard disk not to memory.
> > >>
> > >> Did you look at Tramline as I suggested the last time you  
> asked about
> > >> file uploads?
> > >>
> > >> Tramline is an input filter and also seeks to solve the  
> problem of
> > >> how
> > >> to handle large file uploads.
> > >>
> > >> The URL for Tramline is:
> > >>
> > >>    http://www.infrae.com/download/tramline
> > >>
> > >> Just because Tramline is intended more for cases where Apache is
> > >> front ending a backend application doesn't mean it isn't a valid
> > >> example
> > >> of input filters, you would just need to adapt the ideas to your
> > >> needs.
> > >>
> > >> Graham
> > >

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mm_cfg_has_not_been_edited_to_set_host_domains/pipermail/mod_python/attachments/20061227/4d7da990/attachment-0001.html

More information about the Mod_python mailing list