|
Terence MacDonald
terencemacdonald at gmail.com
Thu Oct 6 07:15:42 EDT 2005
Thanks for the in depth explanation Graham. It HAS helped me understand the problem and has helped me fix it (for now) On a different note I would like to take this opportunity to voice my appreciation to a number of volunteers (you know who you are) that have stepped forward in the past year or so and given mod_python a fresh impetus. I find you extremely helpful and knowledgeable. Hats off to you guys! Roll on 3.2 (3.3...) On 10/5/05, Graham Dumpleton <grahamd at dscpl.com.au> wrote: > > Terence MacDonald wrote .. > > > > On Wed, 2005-10-05 at 07:28 +1000, Graham Dumpleton wrote: > > > The only way I can see there possibly being a problem is if the > > > apache.import_module() method were being used to import > > > pyPgSQL module and it was getting reloaded. I would expect > > > though that pyPgSQL is a standard module in site-packages > > > though so would be imported using "import". Yes/No? > > > > > the PyPgSQL module PgSQL.py is imported using the usual "import", this > > import resides in my database module which itself is imported, by my > > processing modules, using apache.import_module(). I have PythonOption > > AutoReload set to 'on' > > > > I have stopped using apache.import_module inline to import my database > > module when required and resorted to 'top of the file' import instead. > > The problem appears (fingers crossed!) to have gone away. > > > > I am still not sure what the problem or the solution is/was, except that > > it is related to import_module, pickling session data that has object > > instances and a pickling helper function (in this case a function named > > _B that aids in pickling PgBoolean instances and is declared in the > > init.py file of the PyPgSQL package) not being at the same address.... > > If that makes sense > > In short, in mod_python 3.1.4 and earlier, pickling of anything which > depends on a function object which resides in a module imported using > apache.import_module() will not be reliable. With the changes in 3.2, it > simply will not work. > > Lets start with the following tests: > > # >>> import pickle > # >>> def a(): pass > # ... > # >>> pickle.dumps(a) > # 'c__main__\na\np0\n.' > # >>> z = a > # >>> pickle.dumps(z) > # 'c__main__\na\np0\n.' > > As you can see, it is possible to pickle the function object. This can > be done even through a copy of the function object by reference, > although in that case the pickled object still refers to the original > function object. > > Now lets delete the original function object and pickle the copy again. > > # >>> del a > # >>> pickle.dumps(z) > # Traceback (most recent call last): > # ....... deleted traceback > # pickle.PicklingError: Can't pickle <function a at 0x612b0>: it's not > found as __main__.a > > Because the original function object was deleted from where it was > created, one cannot now even pickle the copy. > > Now lets recreate the original function object. > > # >>> def a(): pass > # ... > # >>> pickle.dumps(z) > # Traceback (most recent call last): > # ....... deleted traceback > # pickle.PicklingError: Can't pickle <function a at 0x612b0>: it's not the > same object as __main__.a > > Notice how the exception message has changed. It recognises that "a" > exists but realises that it is actually a different function object from > which the "z" copy was originally made. > > Where problems can start occuring in mod_python 3.1.4 and earlier is if > the function object is cached in some data object which is held outside > of the module the function object was defined in. If the original module > holding the original function object were now reloaded because of the > automatic module reloading mechanism implemented by the > apache.import_module() function, an attempt to pickle the data object > which had cached the function object will fail. This is because the > original function object which had been copied from will have been > overrwritten by a new one when the module was reloaded. > > This sort of problem although it will not occur for an instance of a > class object, will occur for the class object type itself. > > # >>> class B: pass > # ... > # >>> b=B() > # >>> pickle.dumps(b) > # '(i__main__\nB\np0\n(dp1\nb.' > # >>> del B > # >>> pickle.dumps(b) > # '(i__main__\nB\np0\n(dp1\nb.' > > # >>> class B: pass > # ... > # >>> pickle.dumps(B) > # 'c__main__\nB\np0\n.' > # >>> C = B > # >>> pickle.dumps(C) > # 'c__main__\nB\np0\n.' > # >>> del B > # >>> pickle.dumps(C) > # Traceback (most recent call last): > # ........ deleted traceback > # pickle.PicklingError: Can't pickle <class __main__.B at 0x53ab0>: it's > not found as __main__.B > > Thus, in practice, even in mod_python 3.1.4 and earlier I would not > recommend trying to pickle function objects or class object types, > unless you are absolutely gauranteed that the module that the original > function object instance or class object type resides in is only > imported using "import" and is never in anyway reloaded. > > If wanted to ensure that no strange problems were going to occur, I > would possibly go as far as suggesting that only basic Python types, > ie., scalars, tuples, lists and dictionaries, be pickled along with > Session objects. > > One other obscure area that would worry me in respect of pickling and > mod_python is that different parts of a web site can have different > PythonPath settings. The issue here is that when unpickling certain > objects, such as function objects and class object types, the original > module containing a type must be importable within the context that the > unpickling is occuring. This is so that if it hasn't already been > imported it can be automatically imported. > > What though happens when the pickling occurs in one part of the > namespace of a web site and it is unpickled in another where PythonPath > is set differently and the required module hadn't already been imported, > but doesn't appear in any directory specificed by PythonPath. Because of > how mod_python can overlay same named modules over the top of each other > in mod_python 3.1.4, it may also not import the correct module, plus > there is a mixing of the "import" and "apache.import_module()" > mechanisms. > > I may be overly paranoid, but that is what defensive programming is all > about if you really want to ensure you are building a robust system > where you avoid any hint of trouble. :-) > > Anyway, hope this might help in some way to explain your problems. > > Graham > > > > > _______________________________________________ > Mod_python mailing list > Mod_python at modpython.org > http://mailman.modpython.org/mailman/listinfo/mod_python > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mm_cfg_has_not_been_edited_to_set_host_domains/pipermail/mod_python/attachments/20051006/fda24a07/attachment.html
|