[mod_python] Headers not being sent when input/output is over a certain size

On 29/08/2007, Sidnei da Silva <sidnei at enfoldsystems.com> wrote:
> ----- Original Message -----
> > If you expect to be able to set headers in an output filter based on
> > the complete content being output, you need to buffer up all your data
> > and write it in one go after you have updated the headers. You cannot
> > write the data and then update the headers.
>
> That's what I'm doing yes.
>
> > Without seeing the code for your output filter, can't comment further
> > as to what the problem may be.
>
> It goes along the lines of (off the top of my head):
>
> if <request-not-interesting>:
>    filter.pass_on()
>    return
>
> ... do buffering ..
>
> if <read-returns-None>:
>   filter.req.set_content_length(<len-of-generated-data>)
>   filter.req.content_type = 'text/xml'
>   filter.write(<generated-data>)
>   filter.close()
>
> Again, it works as expected with little data, but when there's just a little
> bit over 7.5k of buffered input data, when I call filter.write() the
> Content-Length and Content-Type headers do not get sent.

Do you get a content length header if you enable the Apache
CONTENT_LENGTH output filter?

Part of the problem may be that output filters in Apache are a
resource filter and are only possibly supposed to operate on the
actual body content. I'm not sure if resource filters are supposed to
be able to operate on headers.

The CONTENT_LENGTH output filter for example which does operate on
headers, is registered as a protocol filter and not a resource filter.

I don't have the time to investigate this now, but might be able to
look at what the differences are later and whether you can even set
headers in mod_python output filters.

/**
 * Filters have different types/classifications. These are used to group
 * and sort the filters to properly sequence their operation.
 *
 * The types have a particular sort order, which allows us to insert them
 * into the filter chain in a determistic order. Within a particular grouping,
 * the ordering is equivalent to the order of calls to ap_add_*_filter().
 */
typedef enum {
    /** These filters are used to alter the content that is passed through
     *  them. Examples are SSI or PHP. */
    AP_FTYPE_RESOURCE     = 10,
    /** These filters are used to alter the content as a whole, but after all
     *  AP_FTYPE_RESOURCE filters are executed.  These filters should not
     *  change the content-type.  An example is deflate.  */
    AP_FTYPE_CONTENT_SET  = 20,
    /** These filters are used to handle the protocol between server and
     *  client.  Examples are HTTP and POP. */
    AP_FTYPE_PROTOCOL     = 30,
    /** These filters implement transport encodings (e.g., chunking). */
    AP_FTYPE_TRANSCODE    = 40,
    /** These filters will alter the content, but in ways that are
     *  more strongly associated with the connection.  Examples are
     *  splitting an HTTP connection into multiple requests and
     *  buffering HTTP responses across multiple requests.
     *
     *  It is important to note that these types of filters are not
     *  allowed in a sub-request. A sub-request's output can certainly
     *  be filtered by ::AP_FTYPE_RESOURCE filters, but all of the "final
     *  processing" is determined by the main request. */
    AP_FTYPE_CONNECTION  = 50,
    /** These filters don't alter the content.  They are responsible for
     *  sending/receiving data to/from the client. */
    AP_FTYPE_NETWORK     = 60
} ap_filter_type;

Graham