[mod_python] The new module loader

Dan Eloff dan.eloff at gmail.com
Thu Apr 20 22:29:41 EDT 2006


Neat, it works like you said it would, very nice. I'm not sure how you
managed to reload parent when child changes, I'd like to take a peek at that
later, good job.

__purge__ is called if __clone__ raises and exception, to notify the
original module that it wasn't able to be cloned, and now it's just going to
be dumped without the data being copied over.

I think I figured out what you used the intialized flag for in child.py, it
let's you avoid initializing the parts that get overwritten by __clone__()
in the event that __clone__ is called, afterall, it could be an expensive
operation. I don't understand why you keep a reference in a temporary
location and then lock the object and copy outside of __clone__, why not
lock and copy inside __clone__?

Thanks,
-Dan

On 4/20/06, Graham Dumpleton <grahamd at dscpl.com.au> wrote:
>
> Dan Eloff wrote ..
> > Yay! I've got it now, and it compiled fine. I suspect something that
> will
> > still be a hitch is the need for that __clone__() method you talked
> about.
> > How do I use it, is it documented anywhere, or is there an example
> > __clone__() method I can see?
> >
> > Thanks,
> > -Dan
>
> Note that the __clone__() method is something that I added because of
> a need I saw. I doubt that anyone else has burrowed into the code for
> the importer deep enough yet to understand what it is necessarily all
> about and thus the whole concept hasn't been peer reviewed as such.
> Thus whether it is included at all, or in the form as it currently exists
> is not final.
>
> That said, I have attached a very contrived set of examples of migrating
> data from an old module to a new. It looks messy purely because of
> the different examples and support for multithreading. If a module is
> complicated and you want to be able to support multithreading and
> module reloading, it will be complicated. I've seen a bit of dubious code
> posted in the past which given the right conditions would blow up,
> but without a proper solution have generally refrained from pointing
> out the problems.
>
> Put the attached file in a directory configured for mod_python.publisher
> with a configuration something like:
>
>   PythonInterpreter testing
>   AddHandler mod_python .py
>   PythonHandler mod_python.publisher
>   PythonDebug On
>   PythonAutoReload On
>
> The "testing" interpreter is the one I have the new module importer
> enabled for. Ie., in main Apache configuration, outside of all container
> directives have:
>
>   PythonOption mod_python.future.importer testing
>
> Check that all works by accessing page as "child.py". Once it works,
> copy "child.py" to "parent.py" and add at the start the line:
>
>   import child
>
> Then access "parent.py".
>
> Now for some fun, update modification time on "parent.py" by touching
> it or resaving it. Watch how reload counter increments. Touch it again
> and see how transfer counter increments. Work out from code what
> each of the bits is doing and why certain things are done certain ways.
> If you don't understand, ask.
>
> Now for some tricky stuff. Touch the "child.py" file, but still access
> "parent.py". See how the parent module gets reloaded. Keep touching
> "child.py" if you don't believe what is happening. Also access "child.py"
> from browser as well to see how was also reloaded.
>
> What is happening here is that new module importer doesn't just look
> at page you are accessing, but looks at modules it imports and if the
> child changes, the parent gets reloaded automatically along with the
> child. With this working, you avoid all the horrible problems that exist
> now of having to force reloads by touching all files or restarting Apache.
>
> Finally, note how the reload is occurring even though child was imported
> using "import" statement. This is because underneath it is magically
> mapping to apache.import_module(). Change the "import" from:
>
>   import child
>
> to:
>
>   child = apache.import_module("child")
>
> and you will see it works the same.
>
> The rules are as per that incomplete document I referred you to
> about the module importer.
>
> BTW, you cant totally avoid Apache restarts when you start using
> __clone__() unless you are quite careful that it handles gracefully
> not finding the data in an existing module that it is looking to copy
> across. Anyway, experiment and you will see what I mean.
>
> You might also see if you can work out what __purge__() function
> does by looking at source code. :-)
>
> Hope this provides you with hours of fun.
>
> Graham
>
> > On 4/20/06, Graham Dumpleton <grahamd at dscpl.com.au> wrote:
> > >
> > > Quoting from some of Jim's notes:
> > >    To access the Subversion repositories anonymously, you will need a
> > > Subversion client.
> > >
> > >    Choose the branch you would like to work on and check it out. For
> > > example, to get the
> > >
> > >    current development branch (trunk), use:
> > >
> > >      svn co http://svn.apache.org/repos/asf/httpd/mod_python/trunk
> > > mod_python
> > >
> > > Graham
> > >
> > > On 21/04/2006, at 4:17 AM, Dan Eloff wrote:
> > >
> > > > Excellent, now I just have to figure out how to get it from the
> > > > svn. There's no instructions (that I found) on the mod_python site,
> > > > probably because it's something everybody but me knows.I downloaded
> > > > TortoiseSVN because that seems to be what you need on windows, and
> > > > it when I click on checkout it wants to know the url of the
> > > > repository.
> > > >
> > > > http://svn.apache.org and http://svn.apache.org/httpd/mod_python/
> > > > do not work.
> > > >
> > > > What's the url (and am I even going about this the right way?)
> > > >
> > > > Thanks,
> > > > -Dan
> > > >
> > > > On 4/19/06, Graham Dumpleton <grahamd at dscpl.com.au> wrote: Dan
> > > > Eloff wrote ..
> > > > > It sounds interesting, it might just finally solve all my
> > > > importing woes.
> > > > > How do I enable this UMI?
> > > > >
> > > > > Thanks,
> > > > > -Dan
> > > >
> > > > See:
> > > >
> > > >
> http://issues.apache.org/jira/browse/MODPYTHON-143#action_12370976
> > > >
> > > > Graham
> > > >
> > > > > On 4/19/06, Graham Dumpleton <grahamd at dscpl.com.au> wrote:
> > > > > >
> > > > > > Dan Eloff wrote ..
> > > > > > > I am interested in how this unified module importer works and
> > > > how it
> > > > > > avoids
> > > > > > > these problems.
> > > > > > >
> > > > > > > I'd really appreciate a brief description of it's
> > > > architecture, not
> > > > > that
> > > > > > > you
> > > > > > > usually need all that much encouraging ;)
> > > > > > >
> > > > > > > Thanks,
> > > > > > > -Dan
> > > > > >
> > > > > > Bit busy today to elaborate too much, but you can start by
> > > > reading my
> > > > > > far from complete document on it:
> > > > > >
> > > > > >   http://www.dscpl.com.au/articles/modpython-007.html
> > > > > >
> > > > > > Graham
> > > > > >
> > > > > > > On 4/19/06, Graham Dumpleton <grahamd at dscpl.com.au> wrote:
> > > > > > > >
> > > > > > > >
> > > > > > > > On 20/04/2006, at 8:06 AM, Dan Eloff wrote:
> > > > > > > >
> > > > > > > > > I went digging through the mod_python files and
> > > > discovered how
> > > > > the
> > > > > > > new
> > > > > > > > > publisher loader works (using ModuleCache) and I borrowed
> > > > the idea
> > > > > > > to
> > > > > > > > > import modules for my handler.
> > > > > > > >
> > > > > > > > Note that mod_python 3.3 will like be discarding that
> > > > module loader
> > > > > > > > and the original one and replacing it with a grand unified
> > > > module
> > > > > > > > reloader. :-)
> > > > > > > >
> > > > > > > > > The purpose of ModuleCache was to avoid sys.modules right?
> > > > > > > >
> > > > > > > > Yes.
> > > > > > > >
> > > > > > > > > I'm having some trouble with this, and I eventually
> > > > tracked it
> > > > > down
> > > > > > > to
> > > > > > > > > multiple versions of a dependancy module being loaded.
> > > > > > > >
> > > > > > > > Known issue. See:
> > > > > > > >
> > > > > > > >
> > > > > > > > http://www.dscpl.com.au/articles/
> > > > modpython-003.html#multiple-module-
> > > > > > > > instances
> > > > > > > >
> > > > > > > > > My handler loads one version, with empty module
> > > > variables, while
> > > > > the
> > > > > > > > > actual modules for the pages load different version(s)
> > > > and populate
> > > > > > > > > them with data. The changes are never reflected in the
> > > > version
> > > > > kept
> > > > > > > by
> > > > > > > > > my handler (because they are not the same)
> > > > > > > > >
> > > > > > > > > This all makes sense, and is probably the intended
> > > > effect, but
> > > > > what
> > > > > > > do
> > > > > > > > > you do for modules like Session in mod_python, where
> > > > MemorySession
> > > > > > > > > uses a static class member to store sessions? Wouldn't
> > > > the page
> > > > > > > > > modules each get their own version and thusly prevent
> > > > sessions
> > > > > from
> > > > > > > > > working?
> > > > > > > >
> > > > > > > > Also see:
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > http://www.dscpl.com.au/articles/modpython-003.html#transfer-of-
> > > > module-
> > > > > > > > data
> > > > > > > >
> > > > > > > > > I need to be able to reload modified modules, but also to
> > > > make
> > > > > use
> > > > > > > of
> > > > > > > > > shared modules. These seem to be conflicting needs,
> > > > however. Does
> > > > > > > > > anybody have any inisght into this situation that they
> > > > can share?
> > > > > > > >
> > > > > > > > Only that mod_python 3.3 will hopefully fix a lot of this
> > > > stuff.
> > > > > If
> > > > > > you
> > > > > > > > are prepared to give 3.3 source code out of subversion
> > > > repository
> > > > > a go
> > > > > > > > to see if things work as you expect, let me know and will
> > > > tell you
> > > > > the
> > > > > > > > magic to enable the unified module importer in 3.3 as at
> > > > moment it
> > > > > is
> > > > > > > > still off by default.
> > > > > > > >
> > > > > > > > Note though that even in 3.3, the "transfer of module data"
> > > > issue
> > > > > > still
> > > > > > > > has to be explicitly dealt with in some way. Although not
> > > > confirmed
> > > > > > how
> > > > > > > > this will be done, the new module importer as it stands now
> > > > allows
> > > > > a
> > > > > > > > module to define a __clone__() method which will be called
> > > > and which
> > > > > > > > can be used to transfer data from an old module to a new
> > > > where data
> > > > > > > > needs
> > > > > > > > to be preserved across reloads.
> > > > > > > >
> > > > > > > > As I said, if interested I'll go into it more later.
> > > > > > > >
> > > > > > > > Graham
> > > > > > > >
> > > > > > > >
> > > > > >
> > > >
> > >
> > >
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mm_cfg_has_not_been_edited_to_set_host_domains/pipermail/mod_python/attachments/20060420/dbc7dbe0/attachment-0001.html


More information about the Mod_python mailing list