[mod_python] focus on shared hosting?

On Friday 22 December 2006 08:17, Jim Gallacher wrote:
> Graham Dumpleton wrote:
> > On 22/12/2006, at 11:47 AM, Jeff Hinrichs - DM&T wrote:
> >> John Udell blogged today about django and mod_python.  However the
> >> question he poses is pertinent to mod_python and projects that depend
> >> on mp like TG and Pylons, et al.
> >>
> >> ...
> >>
> >> I know that there are a few hosts that specialize in python based apps
> >> but what can be done to make mp a must have like php and ruby?  What
> >> can be done technically to make mp a no-brainer decision for most web
> >> hosting companies.  Are there really security or resource utilization
> >> issues that can't be over come?  Maybe the next big mp focus should be
> >> on what can be done to make administering/securing a shared mp install
> >> so that it is easier for shared hosts to implement it.
> >
> > Parts of what you are talking about are things I have been thinking
> > about quite a lot lately, which is, possible future directions for the
> > concept of using Python in conjunction with Apache. Note though that I
> > don't mention mod_python when I say this. :-)
> >
> > The first thing one has to realise about people who want to use Python
> > for web programming is that the majority don't actually care what
> > mod_python is. Where they do want to use Apache, they see mod_python
> > merely as a convenient hopping off point. As such, you have frameworks
> > like Django, TurboGears, CherryPy and applications like Trac (latest)
> > and MoinMoin, which don't actually use any of the handlers supplied with
> > mod_python, nor its capabilities to do automatic module reloading,
> > forms, session handling etc.
> >
> > All these frameworks and applications want to do is specify a handler
> > to be called for the response phase, set up the Python module path and
> > possibly pass in a few options. When their handler is called, they then
> > promptly hide all the mod_python bits under a layer of code and the
> > user doesn't see them.
> >
> > Further, where in the past these frameworks and applications built
> > separate adaptors for the various different web servers, they now all
> > tend to support WSGI. What one thus needs is not mod_python, but
> > mod_wsgi. The code needed to implement such a module would actually
> > be quite little in comparison to mod_python. In fact, none of the pure
> > Python code components of mod_python would even be required.
> >
> > So what would a mod_wsgi need to contain. The first thing it needs to
> > be able to do is handle initialisation of Python and creation of
> > interpreter instances. This code would actually be much simpler than in
> > mod_python as mod_python has to do extra stuff like initialising global
> > mutexes for session management, trigger imports of special internal
> > modules when each interpreter instances is created, as well as importing
> > special user modules as well. In mod_wsgi it wouldn't need to do any of
> > that.
> >
> > The next big difference with mod_wsgi in comparison to mod_python
> > would be that except to trigger its own initialisation, the only handler
> > phase it needs to hook and process is the response phase. Because it is
> > not hooking all phases, it would be much more efficient than mod_python
> > as it wouldn't be doing all that extra work for each request of checking
> > for mod_python handlers for each phase.
> >
> > Next area where mod_python does more work than it needs to is with all
> > its configuration and options directives. Ie., PythonPath,
> > PythonAutoReload, PythonDebug, PythonOption. Apache already has a SetEnv
> > directive which can be used to set key value pairs with the values being
> > placed into the req.subprocess_env table object. Since other parts of
> > Apache already deal with doing that, mod_wsgi could just use SetEnv as
> > the means of getting any configuration information, with the
> > req.subprocess_env table also becoming the basis of the WSGI dictionary
> > which gets passed through to any WSGI handler. Even the definition of
> > what the WSGI handler module and function is, could be specified in this
> > way and as such mod_wsgi would not need to define any directives at all,
> > thus eliminating the need for code to handle them.
> >
> > To then setup mod_wsgi to be used, in the Apache configuration you would
> > have:
> >
> >   <Location /some/path>
> >   SetHandler mod_wsgi
> >   SetEnv mod_wsgi.application module::application
> >   SetEnv mod_wsgi.interpreter myapplication
> >   SetEnv mod_wsgi.directory /directory/containing/module
> >   </Location>
> >
> > Now in mod_python when a handler needs to be executed, a call is first
> > made into a dispatch function implemented in Python and it is that which
> > sets up all the environment and then calls the handler function. With
> > WSGI though, because the API for calling a WSGI application is so simple
> > it would be better to implement the dispatch in C code. Thus the C code
> > of mod_wsgi would import the application module and make the calls into
> > the application as appropriate. By doing this you totally eliminate the
> > need for any separate Python code modules needing to be installed and
> > thus get rid of one of the setup problems with mod_python of not being
> > able to find those extra Python modules. This should make it much easier
> > to install.
> >
> > One issue with web hosting environments especially is being able to
> > separate different users web applications. Although one can't have each
> > application run as a separate user, one can at least separate them into
> > distinct interpreters. To specify the interpreter one would use SetEnv
> > to set mod_wsgi.interpreter. Where you have a lot of applications
> > though, you might not want to have to manually set these. Although
> > mod_rewrite is itself a bit heavy weight, one of the things it can do is
> > set the same variables as SetEnv sets based on stuff which appears in
> > the URL. Thus one could with a mod_rewrite rule automatically specify
> > the interpreter name based on the request URL. This is something that
> > mod_python can't even do because it uses a directive for setting the
> > name of the interpreter.
> >
> > One final issue with interpreters is the initialisation of Python and
> > the creation of the interpreter instances. In this area, mod_wsgi could
> > run in one of two modes. The first would be where mod_wsgi is the only
> > Python module installed, ie. mod_python is not installed. In this
> > situation it could perform the initialisation of Python all itself and
> > also handle creation of interpreter instances. The second mode would be
> > where it detects that mod_python is already loaded. In this situation it
> > would simply use the hooks added in mod_python 3.3 for using
> > mod_python's machinery for accessing Python interpreter instances. By
> > being able to operate in these two modes, the module could either exist
> > by itself, but if needed, also co-operate with mod_python. Thus, just
> > because you run mod_wsgi doesn't mean that you couldn't also run
> > mod_wsgi at the same time.
> >
> > If you have followed what I am talking about, and understand mod_python
> > internals and Apache a bit, you will see that the code for mod_wsgi
> > would actually be quite simple and because it doesn't have to wrap any
> > Apache data structures in Python equivalents, it should be quite easy to
> > create a version which is capable of being compiled on Apache 1.3, 2.0
> > or 2.2.
> >
> > As I mention above, most people don't need the full mod_python
> > and thus this would allow all these higher level applications to still
> > be able to be run (under WSGI) even though they are using an older
> > version of Apache. This would also make it much simpler for web hosting
> > services as well, as they can say that they support anything which is
> > WSGI compliant.
> >
> > Now, the idea of mod_wsgi is only one part of what I have been thinking
> > about for future directions of Python with Apache. It is late now though
> > so I'll go into my other ideas in the coming days, that is if I don't now
> > decide to go and finish my mod_wsgi which I already have the basis for in
> > place and thus get diverted. :-)
> >
> > Comments on this part of my plans for global domination most welcome.
>
> Wow, what a good idea. (both mod_wsgi and world domination). Mod_wsgi
> could pretty much be the mod_python-lite that Anthony spoke of earlier.
> Could mod_python be made forward compatible with mod_wsgi? That way a
> could site start with mod_wsgi but then easily switch to mod_python if
> its additional features where required.

Or something in between mod_wsgi and the current mod_python, approaching 
something like mod_perl that tries to do less than mod_python, but still 
maintains hooks to the other phases of the request cycle.  (I have VERY 
LITTLE understandings of the innards of apache, so I don't know what such a 
solution might entail).

Sean