[mod_python] mod_python and mod_wsgi

On 29/07/07, Roger Binns <rogerb at rogerbinns.com> wrote:
> Graham Dumpleton wrote:
> > These include things related to main interpreter simplified
> > GIL state API use, .....  Although,
> > the usefulness of this is limited if your application uses certain
> > third party C extension modules which have not been written properly
> > so as to be able to handle interpreters being recycled.
>
> Can you document somewhere what extension authors are supposed to do?
> Currently we all just follow the main CPython documentation.  Even
> better would be to get the CPython implementation updated (so extension
> authors don't have to change code) or at least updating the CPython
> documentation.

There is nothing wrong with the CPython implementation that needs to
be changed, it is how module writers use it.

The two main issues are (affecting both mod_python and mod_wsgi):

1. Module writers using simplified GIL state API for thread
management. By doing this the module can only be used in the very
first interpreter created by Python. In mod_python this is in the
interpreter selectable by setting PythonInterpreter to
'main_interpreter'. In mod_wsgi one selects this first interpreter by
setting WSGIApplicationGroup to '%{GLOBAL}'. That simplified GIL state
API will only work with first interpreter is I believe documented in
Python documentation.

2. Module writers who do not realise that a C module is only
initialised once per process and cache data which is interpreter
specific from the module initialisation function and then use that
cached data in ways with secondary sub interpreters. In many cases
this will work okay, but in others it will not, such as using
'decimal.Decimal' from main interpreter to create object instances for
secondary sub interpreters. The 'decimal.Decimal' in the secondary sub
interpreters is different due to it being a Python object and not a
CPython object. Thus things like isinstance() fail when comparing
object types. This sort of thing may not be explicitly documented in
Python documentation and since module writers don't contemplate that
their module may be used in situation where there is multiple sub
interpreters may not think about it.

Only other real issue and one that only affects mod_wsgi interpreter
reloading is where an extension module caches references to Python
objects in global C variables but doesn't actually do an Py_INCREF on
it. If the sub interpreter is recycled, that cache Python object can
be destroyed. If new interpreter is created the extension module isn't
reinitialised and that C variable then contains a dangling pointer to
object that has since been destroyed.

I mention these issues to various degrees as being a cause of problems
when using mod_wsgi in the mod_wsgi documentation. If need be, when I
get a chance I'll a specific page on these issues with actual C code
examples.

Graham