[mod_python] Imports, Caching, Handlers and Page Templates.

Graham Dumpleton grahamd at dscpl.com.au
Sun Aug 15 19:22:31 EDT 2004

I'll admit that I have only been looking at mod_python for about a 
month and have only
been on the mailing list for a couple of weeks. Also, I have 
exclusively been playing
with the older 2.7 version of mod_python, as I don't have access to a 
machine which
runs Apache 2.0. All the same, I thought it might be a good time to say 
a few things
about what I have been up to since I have just made available a new 
beta version of a
package I make available and this new version now incorporates some 
specific support
for mod_python.

Okay, see it as a bit of advertising for this package if you want, but 
the mod_python
handler I have come up with can be separated from the package and used 
by itself if
desired. From what I have seen, this mod_python handler provides 
something a bit
different to what is out there, but then I could be wrong, as I haven't 
been looking
too long and I haven't looked at mod_python 3.1 in any real detail yet. 
Thus I may have
missed the obvious and simply reinvented the wheel. Please educate me 
if necessary. :-)

First off, the package is called OSE and it is available from:


The version just made available is OSE 8.0b4.

I will not go into detail about what OSE is all about as it would take 
too long. It is
enough to say that it provides a way of implementing applications 
distributed across
multiple processes or multiple machines. This is done using a 
publish/subscribe and
request/reply messaging system. Although the core is written in C++, 
there are some
Python wrappers provided. As far as mod_python is concerned, the main 
thing I now
provide which would be of interest to people here is a specialised 
mod_python content

Now unless I have missed something, there are two ways to setup 
mod_python to be used.
You can use "AddHandler" to indicate that a specific handler be used 
for all requests
where a specific extension is used, or you can use SetHandler to have 
all requests be
processed by a specific content handler. The two cases are represented 
by mod_python.psp
in the first instance and mod_python.publisher in the latter.

The mod_python handler I have developed uses SetHandler like 
mod_python.publisher, but
whereas mod_python.publisher then does some interpretation of the 
request and translates
it into a totally different calling convention, ie., Python method 
invocation, what I
have created effectively maps requests back onto different basic 
content handlers by
looking at the resource being requested.

In other words, if you had copied the "mptest.py" example file into the 
directory and
made a request against "mptest.html", it would be processed by the 
"handler()" method
contained in "mptest.py". At the same time and in the same directory, 
if there were
also a handler defined in "index.py" and a request was made for 
"index.html" then
the "handler()" method in "index.py" would be used to process the 

Thus, without imposing a particular mechanism for how the resultant 
HTML is generated,
such as with mod_python.psp, you can have multiple basic content 
handlers within the one
directory where which is used is by default determined by stripping off 
the ".html"
extension and replacing it with ".py" to arrive at which handler to 
execute. If it is
determined that for a particular request of a ".html" file there is no 
corresponding ".py"
file, the handler declines to handle the request and thus Apache 
processes it as per
normal. Similarly, if the request was for any resource at all it didn't 
know what to
do with, it would again pass it back to Apache.

Now I would have thought this would have been quite basic functionality 
and why I feel
I must have missed something and that this actually is possible without 
a specialised
content handler. So what have I overlooked?

Anyway, to extend on that, as well as dealing with requests for files 
with a ".html"
extension, it is possible to define that a content handler can produce 
other types of
files as well as, or instead of ".html" files. Thus, you might have in 
a directory a file
called "data.csv". If you were to request this file, the initial 
handler, not knowing
what to do with it, will decline to service the request and instead 
Apache will return
it instead.

What can now be done though, is to add a "data.py" containing a handler 
method. This
isn't going to be used to generate the CSV data, but by defining that 
this handler be
called for ".txt" and ".html" requests, ie., requests against 
"data.txt" and "data.html",
it can be used to translate the data in the CSV file into different 
forms, such as
tab separated and HTML table formatted data.

In summary, if a directory were to contain the files as described 
above, plus an
ordinary HTML file, as listed below:


The result thus of requests would be:

   basic.html - declined by initial handler and content returned by 
   data.csv - declined by initial handler and content returned by Apache
   missing.html - declined by initial handler and Apache returns "Not 
   index.html - initial handler would execute handler in index.py to 
generate content
   mptest.html - initial handler would execute handler in mptest.py to 
generate content
   data.txt - initial handler would execute handler in data.py to 
generate content
   data.html - initial handler would execute handler in data.py to 
generate content

One final case, if the extension which a ".py" file is indicated as 
being able to process
is set to be empty, then that name can be used like a directory with 
additional path
info being passed in to the handler. Ie., if the code file is called 
"proxy.py", using
the URL "proxy/a/b/c" would result in the handler in "proxy.py" being 
executed and
with the path info set to "/a/b/c".

That is the first significant bit of functionality provided. The next 
relates to the
fact that to do this effectively, you need to have a caching system for 
Python code
which is loaded, to avoid reimporting on every request. Caching is done 
by mod_python
itself as well as in systems like mpservlets.

The specialised content handler being supplied actually makes use of a 
module caching
system which OSE has had in it for some time, although it has been 
upgraded a bit
recently and split into a separate file so it can be split out of OSE 
if required.

Unlike mod_python and mpservlets the caching system will reload Python 
code if any
change occurs in the modification time. These other systems only do so 
if the modification
time is newer than before, which is no good if an older version of a 
file is put
back in place of the newer one. Only reloading a file if the 
modification is newer
can also sometimes be problematic when files are shared across the 
network using
something like NFS and machine clocks aren't properly synchronised.

More importantly than that, the module cache system can be used 
explicitly within a
loaded module to load additional modules as children and the cache will 
track the
relationships between the modules and will reload a parent if any of 
the children is
changed. This avoids the problem whereby if you use "import" explicitly 
to load a
common bit of code and you change the common code, you are in most 
cases forced to
restart Apache to throw out the old code remembered by Python so as to 
be able to
pick up the new code.

This mechanism of tracking relationships between loaded modules was 
originally developed
for a servlet based system. Specifically, you might have a base class 
servlet which defines
a lot of site structure and a derived class which defines specific page 
content. As
well as the derived servlet being loaded through the module cache, the 
derived servlet
could load the base class servlet via the cache. Now if the base class 
servlet was
modified, even though the derived class servlet hadn't, if a request 
came in which
meant the derived class servlet were to be executed, the system would 
detect the change
in the base class and automatically reload the derived class which 
would in turn reload
the base class thus picking up the changes in the site structure.

Although originally used on a servlet based system independent of 
mod_python and Apache,
it is still applicable here. The cache also does a few other things to 
avoid unnecessary
reimporting as well as cope with removal of files, but enough said on 
that one.

Next thing which might be of interest is that a wrapper for 
HTMLTemplate is provided
along with a template object caching system for its templates. 
HTMLTemplate is a bit
different to most systems in that there is a quite clear separation 
between HTML and
code used to populate the HTML with data. That is, there is no Python 
code embedded
within the HTML, nor is there necessarily any HTML embedded in the 
Python code. In some
respects it is DOM like, but not really, as to make things more 
efficient it only
indexes and allows you to manipulate nodes marked up in a certain way. 
Since this is
a very poor description of HTMLTemplate, you should simply go to its 
web site for a

As to the wrappers for HTMLTemplate and the template caching system, 
what you can end
up with is:


That is, the HTML is all in "index.html", with necessary node markup 
defined in elements
as necessary. When a request comes in for "index.html", the handler in 
"index.py" is
executed. This loads the HTML for "index.html" via the template cache 
and is given a
copy to work on. It uses the HTMLTemplate API to fill in the data and 
writes back the
rendered result for Apache to to return to the client.

Okay, done for now. If you are interested, get down OSE 8.0b4 and 
specifically look at:


The example has the start of some documentation on how it hangs 
together, but running
out of time before I go on holidays, so wanted to get something out now.

Knowing that most probably will not care much about the larger 
functionality of OSE,
the "examples/apache/README" file describes how to copy the module 
cache and mod_python
handler out of OSE proper into the example so you can run it as is, 
without installing

If you have comments or questions, then post them here on the 
mod_python mailing list
and I'll do my best to address them. If there is enough interest I can 
go on to describe
how I see all this fitting into the larger scheme of things with 
respect to the messaging
system provided with OSE and how one might go about implementing 
sessions and database


Graham Dumpleton (grahamd at dscpl.com.au)

More information about the Mod_python mailing list