|
Daniel Popowich
dpopowich at comcast.net
Mon Jan 24 12:19:46 EST 2005
I agree with Nicolas that this is a "bug". Perhaps not a bug in the
classic sense: something not working as advertised, but a design
issue so gnarly, that it smells like bug.
Thank you Jorey for your detailed email, but I have to disagree with
you when you wrote:
This is consistent with normal Python behaviour, where an
interpreter loads modules from the current directory or
PYTHONPATH. [1]
I came to mod_python after *many* years of intensive python
programming and as a newbie to mod_python was caught by this issue for
days. The way mod_python imports files all at the same root level is
very UN-pythonic. After much hair-pulling I finally realized I needed
to roll my own for mpservlets.
I do think it's possible to patch publisher and keep it backwards
compatible. In mpservlets, I do something like this:
cache = {}
...
fname = full path of source file to be loaded, say, req.filename
...
# check cache
code = cache.get(fname)
if not code:
code = {}
execfile(fname, code)
cache[fname] = code
# search code dictionary for what I'm after, e.g.:
handler = code.get('handler')
if not handler:
raise apache.SERVER_RETURN, apache.http_not_found
handler(req)
You get the idea...
Since publisher should "own" the modules it will import (if well
designed, IMHO, the files exist for mod_python, not general use) and
knows exactly what it's looking for in the modules, I see no reason
why such a caching scheme can't work.
There is an issue of both the python module infrastructure and
mod_python having their own copies of the same module if your
publisher source file is imported by another module, e.g., index.py is
imported by foo.py. With mpservlets I took the path of enforcing
servlets to be in a file that cannot be imported by other modules
(they have to be named .mps); this did two things: 1) prevents the
double caching problem and 2) promotes the separation of interface
code from business logic code.
I also agree with Nicolas that there seems to be a message being sent
when you consider mpservlets, Vampire, his own handler and PSP are all
doing their own importing mechanism. Let's face it, importing in
vanilla python is not the most clear-cut source-inclusion scheme
around (this is the ONLY area where I think java got it right over
python); when you imbed it in a process like apache...ouch...
As for Nicholas' suggestion of removing publisher from the
project...that may be a bit drastic. That being said, I've never
thought it an intuitive place to start for newbies.
But if mpservlets was adopted into the main-stream distro...! :-)
Daniel Popowich
-----------------------------------------------
http://home.comcast.net/~d.popowich/mpservlets/
Nicolas Lehuen writes:
> On Mon, 24 Jan 2005 16:28:47 +0100, Nicolas Lehuen
> <nicolas.lehuen at gmail.com> wrote:
> > On Mon, 24 Jan 2005 09:12:24 -0500, Jorey Bump <list at joreybump.com> wrote:
> > > Nicolas Lehuen wrote:
> > > > On Sun, 23 Jan 2005 18:38:43 -0500, Graham Dumpleton
> > > > <grahamd at dscpl.com.au> wrote:
> > >
> > > >>Package such as mpservlets and Vampire implement their own module
> > > >>importing mechanisms in order to avoid this problem. Ie., with those
> > > >>packages it is safe to have Python code files which are named the same
> > > >>appearing in different directories managed by mod_python running
> > > >>under the same interpreter.
> > > >
> > > > We had a discussion a few months ago about this issue, and I still
> > > > think it's a bug. Like Daniel Popovitch with mpservlets and you with
> > > > Vampire, I had to implement my own import mechanism to work around the
> > > > bug. I think it's time to fix it... I'll add a bug report and see what
> > > > we can do about it.
> > >
> > > I guess I'm on record as considering this to be normal Python behaviour,
> > > and not a bug. Talk of modifying Publisher's current import mechanism
> > > threatens the significant investment I have in the applications I've
> > > already developed. I'd rather see Publisher left alone, as other
> > > frameworks are maturing that address this issue.
> > >
> > > We should be clear about what we are talking about:
> > >
> > > Issue:
> > >
> > > An interpreter imports a module's name into the namespace independent of
> > > its location in the filesystem.
> > >
> > > Pros:
> > >
> > > - This is consistent with normal Python behaviour, where an interpreter
> > > loads modules from the current directory or PYTHONPATH. [1]
> > >
> > > - There is no ambiguity about imports, and no need to rewrite code when
> > > moving the module to another location.
> > >
> > > - The Python language documentation serves as a good resource, as native
> > > behaviours are not contradicted. [2]
> > >
> > > - General purpose code needs *very* little modification to run under
> > > mod_python.publisher.
> > >
> > > - The default Python behaviour of caching imported modules can be used
> > > to great leverage, and plays a significant role in the performance boost
> > > offered by mod_python. [3]
> > >
> > > - With careful planning, per-interpreter package repositories can be
> > > created.
> > >
> > > Cons:
> > >
> > > - This consistently trips up newbies, who are expecting PHP-like
> > > behaviour from their web applications.
> > >
> > > - Within an interpreter (a VirtualHost, for example), such care must be
> > > taken to create unique module names that porting an application or
> > > working in a team environment is significantly more fragile than in
> > > other web technologies, where it usually suffices to throw the code in
> > > its own directory.
> > >
> > > - Although a lot of issues can be addressed in .htaccess files, so much
> > > configuration occurs at the apache level that it is difficult to use
> > > mod_python effectively without apache admin privileges. [4]
> > >
> > > - The Multi-Processing Modules (MPMs) in apache 2 may have a unique
> > > effect on an application. After throwing in the different platforms,
> > > Python versions, and developer styles, it can be very hard to reach a
> > > consensus on whether an issue even exists.
> > >
> > > Possible solution:
> > >
> > > Automatically package module based on the file path.
> > >
> > > Pros:
> > >
> > > - Hopefully, this would provide better separation of modules in namespace.
> > >
> > > Cons:
> > >
> > > - Serious potential to break existing Publisher applications.
> > >
> > > - The same module might be loaded into a separate namespace for
> > > different applications, eliminating the advantage of module caching. [5]
> > >
> > > - This encourages the reuse of module names (such as index.py), which is
> > > considered a bad programming practice in Python. [1]
> > >
> > > - If the code assumes and uses a package name based on its location, it
> > > will need to be rewritten if it is moved.
> > >
> > > - The possibility that autopackaging will arbitrarily create a namespace
> > > clash with a standard module is unacceptably high, especially where
> > > third party packages are installed.
> > >
> > > - It severely limits the ability to test portions of the code from the
> > > command line, because it adds an element of unpredictability to the
> > > module search path.
> > >
> > > - Seasoned Python programmers might consider mod_python.publisher to be
> > > "broken" because it doesn't conform to Python standards. [6]
> > >
> > > [1] For a clear description of Python's module search path, see:
> > > http://www.python.org/doc/current/tut/node8.html#SECTION008110000000000000000
> > >
> > > [2] The documentation for mod_python is sparse because it simply embeds
> > > an interpreter in apache. See the Python documentation for programming
> > > issues: http://www.python.org/doc/current/index.html
> > >
> > > [3] This seems to be the main complaint levied against mod_python, in
> > > one form or the other. I consider it one of it's greatest strengths. For
> > > example, it allows me to import database handling code from a module
> > > once for that interpreter, allowing multiple applications to share the
> > > connection -- a poor man's database pool, if you will.
> > >
> > > [4] The same can be said about php.ini, but noone ever needs to restart
> > > apache to develop PHP applications.
> > >
> > > [5] What is the solution to the problems caused by module caching? It
> > > doesn't make much sense to force mod_python to act like it's spawning a
> > > new interpreter for every request. Run the app as a CGI instead of using
> > > mod_python.
> > >
> > > [6] The strength of Publisher is that it leverages the advantages of
> > > Python with very little overhead, simply adding a number of conveniences
> > > specific to web application development. mod_python.psp is a much better
> > > area to radically depart from this paradigm, as it is expected to
> > > provide PHP-like behaviour.
> >
> > Sorry, I don't have much time, so for now I'll give a short answer : I
> > agree that importing modules should be consistent between the standard
> > Python way (the import keyword) and the apache.import_module way.
> >
> > The problem is that mod_python.publisher does not follow the standard
> > Python way in that it imports published modules as root-level modules.
> > It's as if mod_python.publisher changed sys.path for each and every
> > published module. This is not the standard Python way, and the source
> > for many errors.
> >
> > Note that the current package system in Python is way, way smarter
> > than mod_python.publisher : in a package, you can import fellow
> > modules by giving their local names. It's not recommended by
> > pychecker, but it works. The smart part is that if I have this :
> >
> > /
> > package1.py
> > /subdir
> > __init__.py
> > package1.py
> > package2.py
> >
> > Then whatever I do, /subdir/package1.py is imported as subdir.package1
> > and does not collide with /package1.py. mod_python.publisher is not as
> > smart, this is very confusing and surprising, nearly everybody
> > complains about this and hence, I think this should be filed as a bug.
> >
> > So maybe apache.import_module should not be modified (except for
> > performance, but that's another issue which is already in the bugs
> > database), but mod_python.publisher sure has to be changed.
> >
> > As far as I'm concerned, this bug was sufficient to have me implement
> > a publisher which works the way I want ; Daniel Popovitch and Graham
> > Dumpleton did the same with different designs. If each and every
> > developer develops his own publisher, it may be time to think about
> > changing the built-in one.
> >
> > Regards,
> > Nicolas
> >
>
> Or it could be the sign that mod_python.publisher should be completely
> erased from the project, since it poses much more problems than it
> solves. Maybe we should include both mpservlets and Vampire in
> mod_python, and forget about the default publisher (except maybe for
> compatibility issues).
>
> Regards,
> Nicolas
> _______________________________________________
> Mod_python mailing list
> Mod_python at modpython.org
> http://mailman.modpython.org/mailman/listinfo/mod_python
|