[mod_python] The new module loader

Dan Eloff wrote ..
> The test example you gave using publisher works great. But when I try and
> use that with my current configuration:
> 
> PythonInterpreter testing
> PythonOptimize Off
> PythonDebug On
> PythonAutoReload On
> PythonOption mod_python.future.importer testing
> # don't invoke our python handler if the file has an extension, it doesn't
> handle any addresses like that
> <Directory />
>     SetHandler mod_python
>     PythonHandler PyServer.pyserver

The new module importer completely ignores packages as it is practically
impossible to get any form of automatic module reloading to work
correctly with them when they are more than trivial. As such, packages
are handed off to standard Python __import__ to deal with. That it even
finds the package means that you have it installed in sys.path. Even if
it was a file based module, because it is on sys.path and thus likely to
be installed in a standard location, the new module importer would again
ignore it as it leaves all sys.path modules up to Python __import__
as too dangerous to be mixing importing schemes.

Anyway, that all only applies if you were expecting PyServer.pyserver to
automatically reload upon changes.

> </Directory>
> 
> <LocationMatch "^[^?]*?\.">
>     SetHandler None
> </LocationMatch>
> 
> I'm certain it's not using your importer. Nothing is reloaded when modified.
> Nothing is printed to the log.
> 
> I'm calling apache.import_module(module, 1, 1, path).

With the new module importer, ensure you use:

  apache.import_module(module, path=path)

Do not supply the autoreload and log arguments as they will now
default to None which means that the values will be picked up
automatically from Apache configuration options PythonAutoReload
and PythonDebug. This is mentioned in my document.

Now, if the call to apache.import_module() is at global scope in
PyServer.pyserver, because the package module is not a candidate
for module reloading, that you use apache.import_module() is
not going to do anything. In general, if a package/module is
being imported from sys.path, you shouldn't be using the
apache.import_module() function, at least not at global scope.

A request handler within such a module could use apache.import_module()
as in that case it will be called on each request and the module will
come from the cache if not changed, or will be reloaded from disk if it
has, or isn't in the cache to begin with.

> Any idea why this could be?

Can you describe the structure of your modules and where your
apache.import_module() calls are.

> Does your importer only work with publisher?

Should work for normal handlers as well, with proviso that the handler
file would need to be in document tree directory for automatic
reloading to work from top level.

Graham

> Thanks,
> -Dan
> 
> On 4/20/06, Graham Dumpleton <grahamd at dscpl.com.au> wrote:
> >
> > Dan Eloff wrote ..
> > > Neat, it works like you said it would, very nice. I'm not sure how
> you
> > > managed to reload parent when child changes, I'd like to take a peek
> at
> > > that
> > > later, good job.
> >
> > Lots of magic involved. :-)
> >
> > What it does is keep a dependency tree of imports managed through the
> > importer and determines if parents need reloading based on mtime of
> > child being updated, or if the child had been reimported since the last
> > time the parent was imported.
> >
> > It does this to multiple levels, so you could copy child.py to
> > grandchild.py
> > and add "import grandchild" into "child.py" and you will see that touching
> > grandchild.py will correctly cause both child and parent to be imported
> > when parent accessed.
> >
> > > __purge__ is called if __clone__ raises and exception, to notify the
> > > original module that it wasn't able to be cloned, and now it's just
> > going
> > > to
> > > be dumped without the data being copied over.
> >
> > Yep. Sort of a failsafe. Not sure what you might use it for. Maybe to
> > force
> > close of database connections although you would have to be careful
> > doing that as they may be in use by parallel requests in same process.
> > So, more flag that its all going away and close idle connections and
> > when parallel request release connections they might close automatically.
> >
> > > I think I figured out what you used the intialized flag for in child.py,
> > > it
> > > let's you avoid initializing the parts that get overwritten by
> > __clone__()
> > > in the event that __clone__ is called, afterall, it could be an
> > expensive
> > > operation. I don't understand why you keep a reference in a temporary
> > > location and then lock the object and copy outside of __clone__, why
> not
> > > lock and copy inside __clone__?
> >
> > The temporary was just to show a different variation. There is lots of
> > ways
> > of doing it. The reason for doing the transfer outside of the clone
> > function
> > is that at the clone point the new module hasn't actually been loaded
> and
> > you
> > may need to create instances of classes from the new module to pour the
> > transferred data into. These are only available later.
> >
> > Graham
> >
> > > Thanks,
> > > -Dan
> > >
> > > On 4/20/06, Graham Dumpleton <grahamd at dscpl.com.au> wrote:
> > > >
> > > > Dan Eloff wrote ..
> > > > > Yay! I've got it now, and it compiled fine. I suspect something
> that
> > > > will
> > > > > still be a hitch is the need for that __clone__() method you talked
> > > > about.
> > > > > How do I use it, is it documented anywhere, or is there an example
> > > > > __clone__() method I can see?
> > > > >
> > > > > Thanks,
> > > > > -Dan
> > > >
> > > > Note that the __clone__() method is something that I added because
> of
> > > > a need I saw. I doubt that anyone else has burrowed into the code
> for
> > > > the importer deep enough yet to understand what it is necessarily
> all
> > > > about and thus the whole concept hasn't been peer reviewed as such.
> > > > Thus whether it is included at all, or in the form as it currently
> > exists
> > > > is not final.
> > > >
> > > > That said, I have attached a very contrived set of examples of
> > migrating
> > > > data from an old module to a new. It looks messy purely because of
> > > > the different examples and support for multithreading. If a module
> is
> > > > complicated and you want to be able to support multithreading and
> > > > module reloading, it will be complicated. I've seen a bit of dubious
> > > code
> > > > posted in the past which given the right conditions would blow up,
> > > > but without a proper solution have generally refrained from pointing
> > > > out the problems.
> > > >
> > > > Put the attached file in a directory configured for
> > mod_python.publisher
> > > > with a configuration something like:
> > > >
> > > >   PythonInterpreter testing
> > > >   AddHandler mod_python .py
> > > >   PythonHandler mod_python.publisher
> > > >   PythonDebug On
> > > >   PythonAutoReload On
> > > >
> > > > The "testing" interpreter is the one I have the new module importer
> > > > enabled for. Ie., in main Apache configuration, outside of all
> > container
> > > > directives have:
> > > >
> > > >   PythonOption mod_python.future.importer testing
> > > >
> > > > Check that all works by accessing page as "child.py". Once it works,
> > > > copy "child.py" to "parent.py" and add at the start the line:
> > > >
> > > >   import child
> > > >
> > > > Then access "parent.py".
> > > >
> > > > Now for some fun, update modification time on "parent.py" by touching
> > > > it or resaving it. Watch how reload counter increments. Touch it
> again
> > > > and see how transfer counter increments. Work out from code what
> > > > each of the bits is doing and why certain things are done certain
> > ways.
> > > > If you don't understand, ask.
> > > >
> > > > Now for some tricky stuff. Touch the "child.py" file, but still access
> > > > "parent.py". See how the parent module gets reloaded. Keep touching
> > > > "child.py" if you don't believe what is happening. Also access "
> > child.py"
> > > > from browser as well to see how was also reloaded.
> > > >
> > > > What is happening here is that new module importer doesn't just look
> > > > at page you are accessing, but looks at modules it imports and if
> the
> > > > child changes, the parent gets reloaded automatically along with
> the
> > > > child. With this working, you avoid all the horrible problems that
> > exist
> > > > now of having to force reloads by touching all files or restarting
> > Apache.
> > > >
> > > > Finally, note how the reload is occurring even though child was
> > imported
> > > > using "import" statement. This is because underneath it is magically
> > > > mapping to apache.import_module(). Change the "import" from:
> > > >
> > > >   import child
> > > >
> > > > to:
> > > >
> > > >   child = apache.import_module("child")
> > > >
> > > > and you will see it works the same.
> > > >
> > > > The rules are as per that incomplete document I referred you to
> > > > about the module importer.
> > > >
> > > > BTW, you cant totally avoid Apache restarts when you start using
> > > > __clone__() unless you are quite careful that it handles gracefully
> > > > not finding the data in an existing module that it is looking to
> copy
> > > > across. Anyway, experiment and you will see what I mean.
> > > >
> > > > You might also see if you can work out what __purge__() function
> > > > does by looking at source code. :-)
> > > >
> > > > Hope this provides you with hours of fun.
> > > >
> > > > Graham
> > > >
> > > > > On 4/20/06, Graham Dumpleton <grahamd at dscpl.com.au> wrote:
> > > > > >
> > > > > > Quoting from some of Jim's notes:
> > > > > >    To access the Subversion repositories anonymously, you will
> > need
> > > a
> > > > > > Subversion client.
> > > > > >
> > > > > >    Choose the branch you would like to work on and check it out.
> > > For
> > > > > > example, to get the
> > > > > >
> > > > > >    current development branch (trunk), use:
> > > > > >
> > > > > >      svn co http://svn.apache.org/repos/asf/httpd/mod_python/trunk
> > > > > > mod_python
> > > > > >
> > > > > > Graham
> > > > > >
> > > > > > On 21/04/2006, at 4:17 AM, Dan Eloff wrote:
> > > > > >
> > > > > > > Excellent, now I just have to figure out how to get it from
> the
> > > > > > > svn. There's no instructions (that I found) on the mod_python
> > site,
> > > > > > > probably because it's something everybody but me knows.Idownloaded
> > > > > > > TortoiseSVN because that seems to be what you need on windows,
> > > and
> > > > > > > it when I click on checkout it wants to know the url of the
> > > > > > > repository.
> > > > > > >
> > > > > > > http://svn.apache.org and
> > http://svn.apache.org/httpd/mod_python/
> > > > > > > do not work.
> > > > > > >
> > > > > > > What's the url (and am I even going about this the right way?)
> > > > > > >
> > > > > > > Thanks,
> > > > > > > -Dan
> > > > > > >
> > > > > > > On 4/19/06, Graham Dumpleton <grahamd at dscpl.com.au> wrote:
> Dan
> > > > > > > Eloff wrote ..
> > > > > > > > It sounds interesting, it might just finally solve all my
> > > > > > > importing woes.
> > > > > > > > How do I enable this UMI?
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > > -Dan
> > > > > > >
> > > > > > > See:
> > > > > > >
> > > > > > >
> > > > http://issues.apache.org/jira/browse/MODPYTHON-143#action_12370976
> > > > > > >
> > > > > > > Graham
> > > > > > >
> > > > > > > > On 4/19/06, Graham Dumpleton <grahamd at dscpl.com.au> wrote:
> > > > > > > > >
> > > > > > > > > Dan Eloff wrote ..
> > > > > > > > > > I am interested in how this unified module importer works
> > > and
> > > > > > > how it
> > > > > > > > > avoids
> > > > > > > > > > these problems.
> > > > > > > > > >
> > > > > > > > > > I'd really appreciate a brief description of it's
> > > > > > > architecture, not
> > > > > > > > that
> > > > > > > > > > you
> > > > > > > > > > usually need all that much encouraging ;)
> > > > > > > > > >
> > > > > > > > > > Thanks,
> > > > > > > > > > -Dan
> > > > > > > > >
> > > > > > > > > Bit busy today to elaborate too much, but you can start
> by
> > > > > > > reading my
> > > > > > > > > far from complete document on it:
> > > > > > > > >
> > > > > > > > >   http://www.dscpl.com.au/articles/modpython-007.html
> > > > > > > > >
> > > > > > > > > Graham
> > > > > > > > >
> > > > > > > > > > On 4/19/06, Graham Dumpleton <grahamd at dscpl.com.au> wrote:
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > On 20/04/2006, at 8:06 AM, Dan Eloff wrote:
> > > > > > > > > > >
> > > > > > > > > > > > I went digging through the mod_python files and
> > > > > > > discovered how
> > > > > > > > the
> > > > > > > > > > new
> > > > > > > > > > > > publisher loader works (using ModuleCache) and I
> > borrowed
> > > > > > > the idea
> > > > > > > > > > to
> > > > > > > > > > > > import modules for my handler.
> > > > > > > > > > >
> > > > > > > > > > > Note that mod_python 3.3 will like be discarding that
> > > > > > > module loader
> > > > > > > > > > > and the original one and replacing it with a grand
> > unified
> > > > > > > module
> > > > > > > > > > > reloader. :-)
> > > > > > > > > > >
> > > > > > > > > > > > The purpose of ModuleCache was to avoid sys.modulesright?
> > > > > > > > > > >
> > > > > > > > > > > Yes.
> > > > > > > > > > >
> > > > > > > > > > > > I'm having some trouble with this, and I eventually
> > > > > > > tracked it
> > > > > > > > down
> > > > > > > > > > to
> > > > > > > > > > > > multiple versions of a dependancy module being loaded.
> > > > > > > > > > >
> > > > > > > > > > > Known issue. See:
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > http://www.dscpl.com.au/articles/
> > > > > > > modpython-003.html#multiple-module-
> > > > > > > > > > > instances
> > > > > > > > > > >
> > > > > > > > > > > > My handler loads one version, with empty module
> > > > > > > variables, while
> > > > > > > > the
> > > > > > > > > > > > actual modules for the pages load different version(s)
> > > > > > > and populate
> > > > > > > > > > > > them with data. The changes are never reflected in
> the
> > > > > > > version
> > > > > > > > kept
> > > > > > > > > > by
> > > > > > > > > > > > my handler (because they are not the same)
> > > > > > > > > > > >
> > > > > > > > > > > > This all makes sense, and is probably the intended
> > > > > > > effect, but
> > > > > > > > what
> > > > > > > > > > do
> > > > > > > > > > > > you do for modules like Session in mod_python, where
> > > > > > > MemorySession
> > > > > > > > > > > > uses a static class member to store sessions? Wouldn't
> > > > > > > the page
> > > > > > > > > > > > modules each get their own version and thusly prevent
> > > > > > > sessions
> > > > > > > > from
> > > > > > > > > > > > working?
> > > > > > > > > > >
> > > > > > > > > > > Also see:
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > >
> > http://www.dscpl.com.au/articles/modpython-003.html#transfer-of-
> > > > > > > module-
> > > > > > > > > > > data
> > > > > > > > > > >
> > > > > > > > > > > > I need to be able to reload modified modules, but
> also
> > > to
> > > > > > > make
> > > > > > > > use
> > > > > > > > > > of
> > > > > > > > > > > > shared modules. These seem to be conflicting needs,
> > > > > > > however. Does
> > > > > > > > > > > > anybody have any inisght into this situation that
> they
> > > > > > > can share?
> > > > > > > > > > >
> > > > > > > > > > > Only that mod_python 3.3 will hopefully fix a lot of
> > this
> > > > > > > stuff.
> > > > > > > > If
> > > > > > > > > you
> > > > > > > > > > > are prepared to give 3.3 source code out of subversion
> > > > > > > repository
> > > > > > > > a go
> > > > > > > > > > > to see if things work as you expect, let me know and
> > will
> > > > > > > tell you
> > > > > > > > the
> > > > > > > > > > > magic to enable the unified module importer in 3.3
> as at
> > > > > > > moment it
> > > > > > > > is
> > > > > > > > > > > still off by default.
> > > > > > > > > > >
> > > > > > > > > > > Note though that even in 3.3, the "transfer of module
> > data"
> > > > > > > issue
> > > > > > > > > still
> > > > > > > > > > > has to be explicitly dealt with in some way. Although
> > not
> > > > > > > confirmed
> > > > > > > > > how
> > > > > > > > > > > this will be done, the new module importer as it stands
> > > now
> > > > > > > allows
> > > > > > > > a
> > > > > > > > > > > module to define a __clone__() method which will be
> > called
> > > > > > > and which
> > > > > > > > > > > can be used to transfer data from an old module to
> a new
> > > > > > > where data
> > > > > > > > > > > needs
> > > > > > > > > > > to be preserved across reloads.
> > > > > > > > > > >
> > > > > > > > > > > As I said, if interested I'll go into it more later.
> > > > > > > > > > >
> > > > > > > > > > > Graham
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > >
> > > > > > >
> > > > > >
> > > > > >
> > > >
> > > >
> > > >
> >