|
Graham Dumpleton
grahamd at dscpl.com.au
Thu Apr 20 22:39:54 EDT 2006
Dan Eloff wrote .. > Neat, it works like you said it would, very nice. I'm not sure how you > managed to reload parent when child changes, I'd like to take a peek at > that > later, good job. Lots of magic involved. :-) What it does is keep a dependency tree of imports managed through the importer and determines if parents need reloading based on mtime of child being updated, or if the child had been reimported since the last time the parent was imported. It does this to multiple levels, so you could copy child.py to grandchild.py and add "import grandchild" into "child.py" and you will see that touching grandchild.py will correctly cause both child and parent to be imported when parent accessed. > __purge__ is called if __clone__ raises and exception, to notify the > original module that it wasn't able to be cloned, and now it's just going > to > be dumped without the data being copied over. Yep. Sort of a failsafe. Not sure what you might use it for. Maybe to force close of database connections although you would have to be careful doing that as they may be in use by parallel requests in same process. So, more flag that its all going away and close idle connections and when parallel request release connections they might close automatically. > I think I figured out what you used the intialized flag for in child.py, > it > let's you avoid initializing the parts that get overwritten by __clone__() > in the event that __clone__ is called, afterall, it could be an expensive > operation. I don't understand why you keep a reference in a temporary > location and then lock the object and copy outside of __clone__, why not > lock and copy inside __clone__? The temporary was just to show a different variation. There is lots of ways of doing it. The reason for doing the transfer outside of the clone function is that at the clone point the new module hasn't actually been loaded and you may need to create instances of classes from the new module to pour the transferred data into. These are only available later. Graham > Thanks, > -Dan > > On 4/20/06, Graham Dumpleton <grahamd at dscpl.com.au> wrote: > > > > Dan Eloff wrote .. > > > Yay! I've got it now, and it compiled fine. I suspect something that > > will > > > still be a hitch is the need for that __clone__() method you talked > > about. > > > How do I use it, is it documented anywhere, or is there an example > > > __clone__() method I can see? > > > > > > Thanks, > > > -Dan > > > > Note that the __clone__() method is something that I added because of > > a need I saw. I doubt that anyone else has burrowed into the code for > > the importer deep enough yet to understand what it is necessarily all > > about and thus the whole concept hasn't been peer reviewed as such. > > Thus whether it is included at all, or in the form as it currently exists > > is not final. > > > > That said, I have attached a very contrived set of examples of migrating > > data from an old module to a new. It looks messy purely because of > > the different examples and support for multithreading. If a module is > > complicated and you want to be able to support multithreading and > > module reloading, it will be complicated. I've seen a bit of dubious > code > > posted in the past which given the right conditions would blow up, > > but without a proper solution have generally refrained from pointing > > out the problems. > > > > Put the attached file in a directory configured for mod_python.publisher > > with a configuration something like: > > > > PythonInterpreter testing > > AddHandler mod_python .py > > PythonHandler mod_python.publisher > > PythonDebug On > > PythonAutoReload On > > > > The "testing" interpreter is the one I have the new module importer > > enabled for. Ie., in main Apache configuration, outside of all container > > directives have: > > > > PythonOption mod_python.future.importer testing > > > > Check that all works by accessing page as "child.py". Once it works, > > copy "child.py" to "parent.py" and add at the start the line: > > > > import child > > > > Then access "parent.py". > > > > Now for some fun, update modification time on "parent.py" by touching > > it or resaving it. Watch how reload counter increments. Touch it again > > and see how transfer counter increments. Work out from code what > > each of the bits is doing and why certain things are done certain ways. > > If you don't understand, ask. > > > > Now for some tricky stuff. Touch the "child.py" file, but still access > > "parent.py". See how the parent module gets reloaded. Keep touching > > "child.py" if you don't believe what is happening. Also access "child.py" > > from browser as well to see how was also reloaded. > > > > What is happening here is that new module importer doesn't just look > > at page you are accessing, but looks at modules it imports and if the > > child changes, the parent gets reloaded automatically along with the > > child. With this working, you avoid all the horrible problems that exist > > now of having to force reloads by touching all files or restarting Apache. > > > > Finally, note how the reload is occurring even though child was imported > > using "import" statement. This is because underneath it is magically > > mapping to apache.import_module(). Change the "import" from: > > > > import child > > > > to: > > > > child = apache.import_module("child") > > > > and you will see it works the same. > > > > The rules are as per that incomplete document I referred you to > > about the module importer. > > > > BTW, you cant totally avoid Apache restarts when you start using > > __clone__() unless you are quite careful that it handles gracefully > > not finding the data in an existing module that it is looking to copy > > across. Anyway, experiment and you will see what I mean. > > > > You might also see if you can work out what __purge__() function > > does by looking at source code. :-) > > > > Hope this provides you with hours of fun. > > > > Graham > > > > > On 4/20/06, Graham Dumpleton <grahamd at dscpl.com.au> wrote: > > > > > > > > Quoting from some of Jim's notes: > > > > To access the Subversion repositories anonymously, you will need > a > > > > Subversion client. > > > > > > > > Choose the branch you would like to work on and check it out. > For > > > > example, to get the > > > > > > > > current development branch (trunk), use: > > > > > > > > svn co http://svn.apache.org/repos/asf/httpd/mod_python/trunk > > > > mod_python > > > > > > > > Graham > > > > > > > > On 21/04/2006, at 4:17 AM, Dan Eloff wrote: > > > > > > > > > Excellent, now I just have to figure out how to get it from the > > > > > svn. There's no instructions (that I found) on the mod_python site, > > > > > probably because it's something everybody but me knows.I downloaded > > > > > TortoiseSVN because that seems to be what you need on windows, > and > > > > > it when I click on checkout it wants to know the url of the > > > > > repository. > > > > > > > > > > http://svn.apache.org and http://svn.apache.org/httpd/mod_python/ > > > > > do not work. > > > > > > > > > > What's the url (and am I even going about this the right way?) > > > > > > > > > > Thanks, > > > > > -Dan > > > > > > > > > > On 4/19/06, Graham Dumpleton <grahamd at dscpl.com.au> wrote: Dan > > > > > Eloff wrote .. > > > > > > It sounds interesting, it might just finally solve all my > > > > > importing woes. > > > > > > How do I enable this UMI? > > > > > > > > > > > > Thanks, > > > > > > -Dan > > > > > > > > > > See: > > > > > > > > > > > > http://issues.apache.org/jira/browse/MODPYTHON-143#action_12370976 > > > > > > > > > > Graham > > > > > > > > > > > On 4/19/06, Graham Dumpleton <grahamd at dscpl.com.au> wrote: > > > > > > > > > > > > > > Dan Eloff wrote .. > > > > > > > > I am interested in how this unified module importer works > and > > > > > how it > > > > > > > avoids > > > > > > > > these problems. > > > > > > > > > > > > > > > > I'd really appreciate a brief description of it's > > > > > architecture, not > > > > > > that > > > > > > > > you > > > > > > > > usually need all that much encouraging ;) > > > > > > > > > > > > > > > > Thanks, > > > > > > > > -Dan > > > > > > > > > > > > > > Bit busy today to elaborate too much, but you can start by > > > > > reading my > > > > > > > far from complete document on it: > > > > > > > > > > > > > > http://www.dscpl.com.au/articles/modpython-007.html > > > > > > > > > > > > > > Graham > > > > > > > > > > > > > > > On 4/19/06, Graham Dumpleton <grahamd at dscpl.com.au> wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > On 20/04/2006, at 8:06 AM, Dan Eloff wrote: > > > > > > > > > > > > > > > > > > > I went digging through the mod_python files and > > > > > discovered how > > > > > > the > > > > > > > > new > > > > > > > > > > publisher loader works (using ModuleCache) and I borrowed > > > > > the idea > > > > > > > > to > > > > > > > > > > import modules for my handler. > > > > > > > > > > > > > > > > > > Note that mod_python 3.3 will like be discarding that > > > > > module loader > > > > > > > > > and the original one and replacing it with a grand unified > > > > > module > > > > > > > > > reloader. :-) > > > > > > > > > > > > > > > > > > > The purpose of ModuleCache was to avoid sys.modules right? > > > > > > > > > > > > > > > > > > Yes. > > > > > > > > > > > > > > > > > > > I'm having some trouble with this, and I eventually > > > > > tracked it > > > > > > down > > > > > > > > to > > > > > > > > > > multiple versions of a dependancy module being loaded. > > > > > > > > > > > > > > > > > > Known issue. See: > > > > > > > > > > > > > > > > > > > > > > > > > > > http://www.dscpl.com.au/articles/ > > > > > modpython-003.html#multiple-module- > > > > > > > > > instances > > > > > > > > > > > > > > > > > > > My handler loads one version, with empty module > > > > > variables, while > > > > > > the > > > > > > > > > > actual modules for the pages load different version(s) > > > > > and populate > > > > > > > > > > them with data. The changes are never reflected in the > > > > > version > > > > > > kept > > > > > > > > by > > > > > > > > > > my handler (because they are not the same) > > > > > > > > > > > > > > > > > > > > This all makes sense, and is probably the intended > > > > > effect, but > > > > > > what > > > > > > > > do > > > > > > > > > > you do for modules like Session in mod_python, where > > > > > MemorySession > > > > > > > > > > uses a static class member to store sessions? Wouldn't > > > > > the page > > > > > > > > > > modules each get their own version and thusly prevent > > > > > sessions > > > > > > from > > > > > > > > > > working? > > > > > > > > > > > > > > > > > > Also see: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > http://www.dscpl.com.au/articles/modpython-003.html#transfer-of- > > > > > module- > > > > > > > > > data > > > > > > > > > > > > > > > > > > > I need to be able to reload modified modules, but also > to > > > > > make > > > > > > use > > > > > > > > of > > > > > > > > > > shared modules. These seem to be conflicting needs, > > > > > however. Does > > > > > > > > > > anybody have any inisght into this situation that they > > > > > can share? > > > > > > > > > > > > > > > > > > Only that mod_python 3.3 will hopefully fix a lot of this > > > > > stuff. > > > > > > If > > > > > > > you > > > > > > > > > are prepared to give 3.3 source code out of subversion > > > > > repository > > > > > > a go > > > > > > > > > to see if things work as you expect, let me know and will > > > > > tell you > > > > > > the > > > > > > > > > magic to enable the unified module importer in 3.3 as at > > > > > moment it > > > > > > is > > > > > > > > > still off by default. > > > > > > > > > > > > > > > > > > Note though that even in 3.3, the "transfer of module data" > > > > > issue > > > > > > > still > > > > > > > > > has to be explicitly dealt with in some way. Although not > > > > > confirmed > > > > > > > how > > > > > > > > > this will be done, the new module importer as it stands > now > > > > > allows > > > > > > a > > > > > > > > > module to define a __clone__() method which will be called > > > > > and which > > > > > > > > > can be used to transfer data from an old module to a new > > > > > where data > > > > > > > > > needs > > > > > > > > > to be preserved across reloads. > > > > > > > > > > > > > > > > > > As I said, if interested I'll go into it more later. > > > > > > > > > > > > > > > > > > Graham > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
|