[mod_python] focus on shared hosting?

On 22/12/2006, at 11:47 AM, Jeff Hinrichs - DM&T wrote:

> John Udell blogged today about django and mod_python.  However the
> question he poses is pertinent to mod_python and projects that depend
> on mp like TG and Pylons, et al.
>
> ...
>
> I know that there are a few hosts that specialize in python based apps
> but what can be done to make mp a must have like php and ruby?  What
> can be done technically to make mp a no-brainer decision for most web
> hosting companies.  Are there really security or resource utilization
> issues that can't be over come?  Maybe the next big mp focus should be
> on what can be done to make administering/securing a shared mp install
> so that it is easier for shared hosts to implement it.

Parts of what you are talking about are things I have been thinking
about quite a lot lately, which is, possible future directions for the
concept of using Python in conjunction with Apache. Note though that I
don't mention mod_python when I say this. :-)

The first thing one has to realise about people who want to use Python
for web programming is that the majority don't actually care what
mod_python is. Where they do want to use Apache, they see mod_python
merely as a convenient hopping off point. As such, you have frameworks
like Django, TurboGears, CherryPy and applications like Trac (latest)
and MoinMoin, which don't actually use any of the handlers supplied with
mod_python, nor its capabilities to do automatic module reloading,
forms, session handling etc.

All these frameworks and applications want to do is specify a handler
to be called for the response phase, set up the Python module path and
possibly pass in a few options. When their handler is called, they then
promptly hide all the mod_python bits under a layer of code and the
user doesn't see them.

Further, where in the past these frameworks and applications built
separate adaptors for the various different web servers, they now all
tend to support WSGI. What one thus needs is not mod_python, but
mod_wsgi. The code needed to implement such a module would actually
be quite little in comparison to mod_python. In fact, none of the pure
Python code components of mod_python would even be required.

So what would a mod_wsgi need to contain. The first thing it needs to
be able to do is handle initialisation of Python and creation of interpreter
instances. This code would actually be much simpler than in mod_python
as mod_python has to do extra stuff like initialising global mutexes for
session management, trigger imports of special internal modules when
each interpreter instances is created, as well as importing special user
modules as well. In mod_wsgi it wouldn't need to do any of that.

The next big difference with mod_wsgi in comparison to mod_python
would be that except to trigger its own initialisation, the only handler
phase it needs to hook and process is the response phase. Because it is
not hooking all phases, it would be much more efficient than mod_python
as it wouldn't be doing all that extra work for each request of checking
for mod_python handlers for each phase.

Next area where mod_python does more work than it needs to is with all
its configuration and options directives. Ie., PythonPath, PythonAutoReload,
PythonDebug, PythonOption. Apache already has a SetEnv directive which
can be used to set key value pairs with the values being placed into the
req.subprocess_env table object. Since other parts of Apache already deal
with doing that, mod_wsgi could just use SetEnv as the means of getting
any configuration information, with the req.subprocess_env table also
becoming the basis of the WSGI dictionary which gets passed through
to any WSGI handler. Even the definition of what the WSGI handler module
and function is, could be specified in this way and as such mod_wsgi would
not need to define any directives at all, thus eliminating the need for code
to handle them.

To then setup mod_wsgi to be used, in the Apache configuration you would
have:

  <Location /some/path>
  SetHandler mod_wsgi
  SetEnv mod_wsgi.application module::application
  SetEnv mod_wsgi.interpreter myapplication
  SetEnv mod_wsgi.directory /directory/containing/module
  </Location>

Now in mod_python when a handler needs to be executed, a call is first
made into a dispatch function implemented in Python and it is that which
sets up all the environment and then calls the handler function. With
WSGI though, because the API for calling a WSGI application is so simple
it would be better to implement the dispatch in C code. Thus the C code
of mod_wsgi would import the application module and make the calls into
the application as appropriate. By doing this you totally eliminate the
need for any separate Python code modules needing to be installed and
thus get rid of one of the setup problems with mod_python of not being
able to find those extra Python modules. This should make it much easier
to install.

One issue with web hosting environments especially is being able to
separate different users web applications. Although one can't have each
application run as a separate user, one can at least separate them into
distinct interpreters. To specify the interpreter one would use SetEnv
to set mod_wsgi.interpreter. Where you have a lot of applications
though, you might not want to have to manually set these. Although
mod_rewrite is itself a bit heavy weight, one of the things it can do is
set the same variables as SetEnv sets based on stuff which appears in
the URL. Thus one could with a mod_rewrite rule automatically specify
the interpreter name based on the request URL. This is something that
mod_python can't even do because it uses a directive for setting the
name of the interpreter.

One final issue with interpreters is the initialisation of Python and
the creation of the interpreter instances. In this area, mod_wsgi could
run in one of two modes. The first would be where mod_wsgi is the only
Python module installed, ie. mod_python is not installed. In this
situation it could perform the initialisation of Python all itself and
also handle creation of interpreter instances. The second mode would be
where it detects that mod_python is already loaded. In this situation it
would simply use the hooks added in mod_python 3.3 for using
mod_python's machinery for accessing Python interpreter instances. By
being able to operate in these two modes, the module could either exist
by itself, but if needed, also co-operate with mod_python. Thus, just
because you run mod_wsgi doesn't mean that you couldn't also run
mod_wsgi at the same time.

If you have followed what I am talking about, and understand mod_python
internals and Apache a bit, you will see that the code for mod_wsgi
would actually be quite simple and because it doesn't have to wrap any
Apache data structures in Python equivalents, it should be quite easy to
create a version which is capable of being compiled on Apache 1.3, 2.0
or 2.2.

As I mention above, most people don't need the full mod_python
and thus this would allow all these higher level applications to still
be able to be run (under WSGI) even though they are using an older
version of Apache. This would also make it much simpler for web hosting
services as well, as they can say that they support anything which is
WSGI compliant.

Now, the idea of mod_wsgi is only one part of what I have been thinking
about for future directions of Python with Apache. It is late now though so
I'll go into my other ideas in the coming days, that is if I don't now decide
to go and finish my mod_wsgi which I already have the basis for in place
and thus get diverted. :-)

Comments on this part of my plans for global domination most welcome.

Graham