[mod_python] focus on shared hosting?

Some delayed comments on some old posts.

On 23/12/2006, at 12:37 AM, Sean Davis wrote:

> On Friday 22 December 2006 08:17, Jim Gallacher wrote:
>> Graham Dumpleton wrote:
>>> If you have followed what I am talking about, and understand  
>>> mod_python
>>> internals and Apache a bit, you will see that the code for mod_wsgi
>>> would actually be quite simple and because it doesn't have to  
>>> wrap any
>>> Apache data structures in Python equivalents, it should be quite  
>>> easy to
>>> create a version which is capable of being compiled on Apache  
>>> 1.3, 2.0
>>> or 2.2.
>>>
>>> As I mention above, most people don't need the full mod_python
>>> and thus this would allow all these higher level applications to  
>>> still
>>> be able to be run (under WSGI) even though they are using an older
>>> version of Apache. This would also make it much simpler for web  
>>> hosting
>>> services as well, as they can say that they support anything  
>>> which is
>>> WSGI compliant.
>>>
>>> Now, the idea of mod_wsgi is only one part of what I have been  
>>> thinking
>>> about for future directions of Python with Apache. It is late now  
>>> though
>>> so I'll go into my other ideas in the coming days, that is if I  
>>> don't now
>>> decide to go and finish my mod_wsgi which I already have the  
>>> basis for in
>>> place and thus get diverted. :-)
>>>
>>> Comments on this part of my plans for global domination most  
>>> welcome.
>>
>> Wow, what a good idea. (both mod_wsgi and world domination). Mod_wsgi
>> could pretty much be the mod_python-lite that Anthony spoke of  
>> earlier.
>> Could mod_python be made forward compatible with mod_wsgi? That way a
>> could site start with mod_wsgi but then easily switch to  
>> mod_python if
>> its additional features where required.

People run WSGI stuff on mod_python now, so upgrade path shouldn't be a
problem as the handler acting as the gateway would hide any differences
in underlying API which is effectively the whole point of WSGI, ie.,  
make it
run anywhere. That said, how one configures the handler for each may  
need
to be a bit different.

> Or something in between mod_wsgi and the current mod_python,  
> approaching
> something like mod_perl that tries to do less than mod_python, but  
> still
> maintains hooks to the other phases of the request cycle.  (I have  
> VERY
> LITTLE understandings of the innards of apache, so I don't know  
> what such a
> solution might entail).

It is interesting that you see mod_perl lying between mod_wsgi (a cut  
down
mod_python) and mod_python. In fact mod_perl allows a lot more to be  
done
than mod_python in that it exposes all the Apache innards and API to  
view.
With mod_python it is providing a restricted API that hides most of what
Apache provides and with the API not being similar at all. This  
approach and
the fact that the mod_python API on top of Apache is all explicitly  
coded is
quite limiting and makes maintenance and extension of mod_python a pain.

As to whether it is possible to implement a mod_perl like module for  
Python,
it is, it can work and it is simpler than one might expect as SWIG  
works quite
well for this. I already have basic bindings working as proof of  
concept.

Working out what to do with these new Python bindings for Apache is the
problem as it isn't necessarily straight forward to see how they  
might fit into
the future of mod_python. The main problem is that mod_python got there
first and has the market share. Even if one brought out a new module for
using Python with Apache which is technically a better and more  
efficient
solution, it is going to be hard to dislodge mod_python. Thus, having  
a new
module which requires you can't run the old one is probably impractical.

What are the solutions to this. The first is to acknowledge that  
mod_python got
there first and for any new module to recognise when mod_python is also
running and for the new module to utilise the Python interpreter  
initialisation
and creation code in mod_python through Apache hook functions. This is
to a degree now possible with mod_python 3.3.0b, however as I have  
learnt
since I added that ability to mod_python, the whole way mod_python  
handles
Python initialisation is far from optimal and would impose certain  
limitations
on any new module. Thus, for this to work effectively, the lower  
level layer of
mod_python needs to be reworked so that Python initialisation is done at
a more appropriate time and in a more modular way so that separate  
modules
can harness it better.

So, in this approach, if the new module is the only one loaded, it  
would do
all the initialisation itself. If mod_python is loaded as well, it  
would defer to
mod_python as the incumbent system. A problem with this though is  
that all
processing is then going to be weighed down with the checking done by
both modules in every phase. So, although it may be good in that it  
allows
both to run, it is unlikely that people would want to run both and so  
the
incumbent will probably still win.

The second option is to implement the new module as a broad enough
framework that the existing mod_python API could be implemented as a
personality on top of it. So, one looks at this other way around and  
says that
the new module is technically a much better solution and bless it as  
the way
forward. The API of mod_python could then be supported as just one of  
many
restrictive APIs on top of the new module, thus allowing support of  
older code
as well as allowing development of new personalities and extension  
modules
for Apache. As a further example, WSGI could be just another personality
on top of the new module.

With a new module which exposes better the Apache APIs, examples of
things you could do are:

   mod_authn_python - Implement an auth provider using Python code.

   mod_dav_python - Implement a dav provider using Python code.

One could also effectively break up the monolithic feature set of  
mod_python
into smaller bits, thereby allowing users to choose the bits they  
need thereby
limiting the overhead on the server of the bits they don't want. As  
such, one
could have:

   mod_python_handlers - Implementation of hooks for handler phases.

   mod_python_filters - Implementation of hooks for input/output  
filters.

   mod_include_python - Implementation of Python code in server side  
include.

Although technically this solution of implementing the existing  
mod_python
API on top of a new module may be a good way of doing it, I feel that  
it is
also doomed to failed, this time because of politics and the  
unwillingness of
people to accept that such a drastic change is even required. There  
are much
lesser issues related to mod_python already which decisions can't  
even be
made about, so hardly expect any agreement on any sort of future  
direction
for mod_python along these lines.

So, although I may have grand plans for world domination along these  
lines,
in practice I don't see it going very far, at least not under the  
umbrella of the
existing mod_python project. What exactly I will do with these ideas  
and the
code I have already developed I don't know yet. I see I have two  
choices at
present. These are to work on it myself out of my own site as a personal
project purely for curiosity value. The second is to get an Apache  
labs project
created for it and work on it there. In the latter, because it is  
under the
umbrella of the Apache Software Foundation, it doesn't rule out it  
someday
perhaps being adopted as an official front line project. Doing it as  
an Apache
labs project would also make it easier to migrate at least some of  
the ideas over
to mod_python so as to improve the existing code base.

BTW, to give you an idea of what overheads/inefficiencies exist in the
mod_python code base I have done some comparative tests between a
C coded handler module, a mod_python handler function and a handler
function implemented using Apache SWIG bindings on a more streamlined
Python module for Apache that doesn't rely on mod_python.

On my quite slow PowerPC laptop the results I get for a basic hello  
world
response handler written entirely in C code averages about 1250 requests
per second.

For my cut down Python module invoking Python handler which uses SWIG
bindings it averages about 800 requests per second.

With mod_python it averages about 500 requests per second.

Thus, it would appear that there is certainly some room for  
improvement with
mod_python at least and if nothing else, maybe what I am doing might  
help
to identify the problems it has and make it run better. Other than  
that, I don't
know at this point.

Graham