[mod_python] Global Interpreter Lock problem with Berkeley DB XML and mod_python

Graham Dumpleton graham.dumpleton at gmail.com
Wed Apr 11 18:16:42 EDT 2007


Can you point me to the relevant files and sections of files where the
callback from C code where no GIL is held, is made into Python code.
If I can see how that is being done and understand it, may be able to
suggest something.

Graham

On 12/04/07, George Feinberg <george.feinberg at oracle.com> wrote:
>
> Graham,
>
> This is George Feinberg, and I am responsible for BDB XML development,
> and I happened to make the change to use -threads in SWIG generation.
>
> Perhaps you can help -- you seem to know more about it than I do.
> Here's
> the history:
>
> David's using release 2.3.10, where -threads was added.
> In release 2.2.13 (the previous one), it was not used, but there was
> a problem.
> Multi-threaded Python programs, they would deadlock themselves on
> a combination of BDB XML locking and the GIL.  One of the threads's
> stacks
> looked a bit like this:
>
> #6  0x080dd045 in PyThread_acquire_lock ()
> #7  0x080b1141 in PyEval_EvalFrame ()
> #8  0x080b726d in PyEval_EvalCodeEx ()
> #9  0x080b5719 in PyEval_EvalFrame ()
> #10 0x080b580c in PyEval_EvalFrame ()
> #11 0x080b726d in PyEval_EvalCodeEx ()
> #12 0x080fb6fd in PyFunction_SetClosure ()
> #13 0x0805950c in PyObject_Call ()
> #14 0x0805eac5 in PyMethod_New ()
> #15 0x0805950c in PyObject_Call ()
> #16 0x0808a965 in _PyType_Lookup ()
> #17 0x0808ca88 in _PyObject_SlotCompare ()
> #18 0x0805950c in PyObject_Call ()
> #19 0x080afed7 in PyEval_CallObjectWithKeywords ()
> #20 0x080594e3 in PyObject_CallObject ()
> #21 0xb7b0b360 in SWIG_Python_NewPointerObj (ptr=Variable "ptr" is not
> available.
> ) at dbxml_python_wrap.cpp:1266
> #22 0xb7b32df6 in _wrap_XmlManager_query__SWIG_3 (args=Variable
> "args" is not
> available.
> ) at dbxml_python_wrap.cpp:6675
> #23 0xb7b3332a in _wrap_XmlManager_query (self=0x0, args=0xb6baeb1c) at
>
> There was another thread involved, and it was blocked inside BDB XML
> waiting
> for a page lock held by the thread above.  Presumably it also held
> the lock
> (GIL) that caused this thread to block.
>
> So... I tried a bunch of things to fix this, but the fact that python
> 2.3 and 2.4 have
> different lock-related APIs and you crash in 2.3 if you use the wrong
> one didn't help.
> Turning on -threads, plus some other custom code did the trick.
>
> I'm perfectly happy with another solution if there is one.
> Eliminating Python 2.3
> support, or at least making it conditional it might help.
> Perhaps you have a suggestion.  I'm not a Python expert by any
> stretch of the imagination.
> I ended up here via hacking.
>
> There are a couple of things that I'm *trying* to accomplish in the
> module:
> 1.  release the GIL while in BDB XML to allow other threads to run in
> Python.
> 2.  the module uses SWIG Directors, so calls back into those (from C+
> +) need to
> reacquire the lock, or things go badly.
>
> What is the best way to do this?  Just eliminating -threads will
> cause things
> to appear to be working until they deadlock in the non-mod_python
> environment.   I'm guessing that using your
> active state threads may deadlock too, but I'm not certain.
>
> Thanks for any advice,
>
> George
>
> On Apr 10, 2007, at 11:06 PM, Graham Dumpleton wrote:
>
> > FWIW, am not sure why the -threads option is used as I can't see that
> > it would be required when the package is used exclusively from Python
> > code. The only time one would potentially need the GIL wrappers that
> > SWIG puts in there is when some C code which doesn't currently have
> > the Python GIL held or an active thread state makes a call to one of
> > the generated SWIG wrapper functions.
> >
> > Unless the dbxml package has been specially designed to be used in
> > embedded systems where C code is calling out of C code into
> > dbxml/Python code and the calling C code hasn't performed any prior
> > Python initialisation steps, this is unlikely to occur. Only other
> > possibility is that dbxml specifically has sections where it releases
> > the GIL but then wants to be able to perform a callback into Python
> > code, but I can't see any instances of Py_BEGIN_ALLOW_THREADS and
> > Py_END_ALLOW_THREADS being used in the code.
> >
> > The reason the GIL wrappers cause a problem for mod_python is that
> > mod_python has already correctly acquired the GIL and created an
> > active thread state, but such thread states are not registered with
> > Python's simplified GIL API system because one is not meant to use the
> > Python simplified GIL API system when multiple Python sub interpreters
> > are used for various reasons. It says as much in the Python code and
> > also I think the documentation.
> >
> > Not sure how much joy you will get with getting dbxml changed. Really
> > need to find out exactly why dbxml authors think the -threads option
> > is required.
> >
> > Graham
> >
> >> Graham Dumpleton wrote:
> >> > Looking a bit further, the GIL macros are in my own stuff, but I
> >> > suspect it doesn't get enabled because I don't use the -threads
> >> option
> >> > to SWIG. I don't understand the consequences of the option, but
> >> > changing:
> >> >
> >> >  dbxml/dist/s_swig
> >> >
> >> > so that '-threads' isn't in swig_args for Python would probably get
> >> > rid of the stuff.
> >> >
> >> > Would need to work out why they use the option in the first
> >> place as
> >> > turning it off could break things if dbxml is dependent on it in
> >> some
> >> > way.
> >> >
> >> > Graham
> >> >
> >> > On 11/04/07, Graham Dumpleton <graham.dumpleton at gmail.com> wrote:
> >> >> Try rebuilding dbxml but ensure that compiler option:
> >> >>
> >> >>   -DSWIG_PYTHON_NO_USE_GIL
> >> >>
> >> >> is used.
> >> >>
> >> >> I don't know when SWIG changed, it certainly didn't generate these
> >> >> PyGILState calls the last time I played with it, which wasn't that
> >> >> long ago, but it isn't going to work under mod_python the way
> >> they now
> >> >> do it as the default.
> >> >>
> >> >> I'll have to look into this further when I get a chance as this
> >> new
> >> >> behaviour of SWIG is going to cause all my Apache API SWIG
> >> bindings to
> >> >> break. :-(
> >> >>
> >> >> Graham
> >> >>
> >> >> On 11/04/07, David Schachter <ds at bittorrent.com> wrote:
> >> >> >
> >> >> >  Folks--
> >> >> >
> >> >> >  I'm having a problem embedding Berkeley DB XML into Apache with
> >> >> mod_python.
> >> >> > The first attempt to create an XmlManager causes a lockup in
> >> >> Python--the
> >> >> > Global Interpreter Lock is deadlocked, I think. (Stack trace
> >> >> below.) Any
> >> >> > ideas of how to debug? Is there a BDB XML Python expert? I
> >> checked
> >> >> Google
> >> >> > without success.
> >> >> >
> >> >> >  The code to demonstrate the problem is simple and shown below.
> >> >> When the
> >> >> > line "xmlManager = dbxml.XmlManager()" is commented out, the
> >> code
> >> >> runs ok.
> >> >> > Commented in, Apache locks up.
> >> >> >
> >> >> >  The most recent mod_python has this comment at
> >> >> >
> >> >> http://www.modpython.org/live/mod_python-3.3.1/doc-html/app-
> >> changes.html:
> >> >>
> >> >> >
> >> >> >
> >> >> > Third party C modules that use the simplified API for the Global
> >> >> Interpreter
> >> >> > Lock (GIL), as described in PEP 311, can now be used. The only
> >> >> requirement
> >> >> > is that such modules can only be used in the context of the
> >> >> > "main_interpreter". (More info at
> >> >> > http://issues.apache.org/jira/browse/MODPYTHON-77.)
> >> >> >  The documentation for mod_python at
> >> >> > http://www.modpython.org/live/current/doc-html/pyapi-
> >> interps.html
> >> >> > says:
> >> >> >
> >> >> >
> >> >> > Note that if any third party module is being used which has a
> >> C code
> >> >> > component that uses the simplified API for access to the Global
> >> >> Interpreter
> >> >> > Lock (GIL) for Python extension modules, then the interpreter
> >> name
> >> >> must be
> >> >> > forcibly set to be "main_interpreter". This is necessary as
> >> such a
> >> >> module
> >> >> > will only work correctly if run within the context of the
> >> first Python
> >> >> > interpreter created by the process.
> >> >> >  Conforming to this, I set the interpreter name in the
> >> httpd.conf
> >> >> file using
> >> >> > the directive "PythonInterpreter main_interpreter" and I
> >> verified
> >> >> this by
> >> >> > printing the interpreter name from my Python code.
> >> >> >
> >> >> >  In dbxml-2.3.10/dbxml/src/python, there's some code relating to
> >> >> the GIL in
> >> >> > dbxml_python_wrap.cpp. If someone else has already beaten this
> >> >> problem into
> >> >> > submission, I'd rather not duplicate the effort.
> >> >> >
> >> >> >                           -- David Schachter
> >> >> >
> >> >> >  #! /usr/bin/python
> >> >> >
> >> >> >  import dbxml
> >> >> >  from mod_python import apache
> >> >> >  import os
> >> >> >
> >> >> >
> >> >> >  def handler(req):
> >> >> >    id = "pid %d, interpreter '%s'" % (os.getpid(),
> >> req.interpreter)
> >> >> >
> >> >> >    apache.log_error("About to start BDB XML from %s." % id)
> >> >> >    xmlManager = dbxml.XmlManager()
> >> >> >    apache.log_error("Started BDB XML.")
> >> >> >
> >> >> >    req.write("Hi from %s." % id)
> >> >> >    return apache.OK
> >> >> >
> >> >> >
> >> >> >  STACK TRACE IS:
> >> >> >
> >> >> >  (gdb) bt
> >> >> >  #0  0xffffe410 in __kernel_vsyscall ()
> >> >> >  #1  0xb7e97af0 in sem_wait at GLIBC_2.0 () from
> >> >> > /lib/tls/i686/cmov/libpthread.so.0
> >> >> >  #2  0x00000004 in ?? ()
> >> >> >  #3  0xb7ce7a00 in PyThread_acquire_lock (lock=0x0,
> >> waitflag=1) at
> >> >> > thread_pthread.h:298
> >> >> >  #4  0xb7cb5689 in PyEval_RestoreThread (tstate=0x810f888) at
> >> >> > Python/ceval.c:305
> >> >> >  #5  0xb7ce16e7 in PyGILState_Ensure () at Python/pystate.c:512
> >> >> >  #6  0xb78282f9 in _wrap_new_XmlManager (self=0x0,
> >> args=0xb7b9102c) at
> >> >> > dbxml_python_wrap.cpp:898
> >> >> >       (The reference to line 898 is correct but surprising, the
> >> >> result of a
> >> >> > macro expansion at line 4195. --D.S.)
> >> >> >  #7  0xb7c65967 in PyObject_Call (func=0x0, arg=0xfffffffc,
> >> >> kw=0xfffffffc)
> >> >> > at Objects/abstract.c:1795
> >> >> >  #8  0xb7cb9e1d in PyEval_EvalFrame (f=0x822cdbc) at
> >> >> Python/ceval.c:3845
> >> >> >  #9  0xb7cbd719 in PyEval_EvalCodeEx (co=0xb6b12e20,
> >> >> globals=0xfffffffc,
> >> >> > locals=0x0, args=0x0, argcount=1,
> >> >> >      kws=0x0, kwcount=0, defs=0x0, defcount=0, closure=0x0) at
> >> >> > Python/ceval.c:2741
> >> >> >  #10 0xb7d0d6be in function_call (func=0xb6b0902c,
> >> arg=0xb6b892ac,
> >> >> kw=0x0)
> >> >> > at Objects/funcobject.c:548
> >> >> >  #11 0xb7c65967 in PyObject_Call (func=0x0, arg=0xfffffffc,
> >> >> kw=0xfffffffc)
> >> >> > at Objects/abstract.c:1795
> >> >> >  #12 0xb7c6e389 in instancemethod_call (func=0xfffffffc,
> >> >> arg=0xb6b892ac,
> >> >> > kw=0xfffffffc)
> >> >> >      at Objects/classobject.c:2532
> >> >> >  #13 0xb7c65967 in PyObject_Call (func=0x0, arg=0xfffffffc,
> >> >> kw=0xfffffffc)
> >> >> > at Objects/abstract.c:1795
> >> >> >  #14 0xb7c99728 in slot_tp_init (self=0xb6af1fac,
> >> args=0xfffffffc,
> >> >> > kwds=0xfffffffc)
> >> >> >      at Objects/typeobject.c:4774
> >> >> >  #15 0xb7c91ad4 in type_call (type=0xb7c99675, args=0xb7b9102c,
> >> >> kwds=0x0) at
> >> >> > Objects/typeobject.c:435
> >> >> >  #16 0xb7c65967 in PyObject_Call (func=0x0, arg=0xfffffffc,
> >> >> kw=0xfffffffc)
> >> >> > at Objects/abstract.c:1795
> >> >> >  #17 0xb7cba239 in PyEval_EvalFrame (f=0x820f224) at
> >> >> Python/ceval.c:3776
> >> >> >  #18 0xb7cbd0eb in PyEval_EvalFrame (f=0x8195854) at
> >> >> Python/ceval.c:3651
> >> >> >  #19 0xb7cbd0eb in PyEval_EvalFrame (f=0x81c9ac4) at
> >> >> Python/ceval.c:3651
> >> >> >  #20 0xb7cbd719 in PyEval_EvalCodeEx (co=0xb6b87e20,
> >> >> globals=0xfffffffc,
> >> >> > locals=0x0, args=0x7, argcount=0,
> >> >> >      kws=0x8282440, kwcount=7, defs=0x0, defcount=0,
> >> closure=0x0) at
> >> >> > Python/ceval.c:2741
> >> >> >  #21 0xb7cbb46a in PyEval_EvalFrame (f=0x82822a4) at
> >> >> Python/ceval.c:3660
> >> >> >  #22 0xb7cbd719 in PyEval_EvalCodeEx (co=0xb6b87ee0,
> >> >> globals=0xfffffffc,
> >> >> > locals=0x0, args=0x0, argcount=2,
> >> >> >      kws=0x0, kwcount=0, defs=0x0, defcount=0, closure=0x0) at
> >> >> > Python/ceval.c:2741
> >> >> >  #23 0xb7d0d6be in function_call (func=0xb6af3a74,
> >> arg=0xb6b7c78c,
> >> >> kw=0x0)
> >> >> > at Objects/funcobject.c:548
> >> >> >  #24 0xb7c65967 in PyObject_Call (func=0x0, arg=0xfffffffc,
> >> >> kw=0xfffffffc)
> >> >> > at Objects/abstract.c:1795
> >> >> >  #25 0xb7c6e389 in instancemethod_call (func=0xfffffffc,
> >> >> arg=0xb6b7c78c,
> >> >> > kw=0xfffffffc)
> >> >> >      at Objects/classobject.c:2532
> >> >> >  #26 0xb7c65bf6 in PyObject_CallMethod (o=0x0,
> >> name=0xb6b891ec "\001",
> >> >> > format=0xb6b891ec "\001")
> >> >> >      at Objects/abstract.c:1795
> >> >> >  #27 0xb7c5f505 in python_handler (req=0x828a7b8,
> >> phase=0xb7d16e59
> >> >> > "PythonHandler") at mod_python.c:1652
> >> >> >  #28 0x08073daf in ap_run_handler (r=0x828a7b8) at config.c:157
> >> >> >  #29 0x080741b1 in ap_invoke_handler (r=0x828a7b8) at
> >> config.c:372
> >> >> >  ---Type <return> to continue, or q <return> to quit---
> >> >> >  #30 0x0808ba77 in ap_process_request (r=0x828a7b8) at
> >> >> http_request.c:258
> >> >> >  #31 0x0808929b in ap_process_http_connection (c=0x8185638) at
> >> >> > http_core.c:184
> >> >> >  #32 0x0807a599 in ap_run_process_connection (c=0x8185638) at
> >> >> > connection.c:43
> >> >> >  #33 0x0809e55a in child_main (child_num_arg=-4) at prefork.c:
> >> 640
> >> >> >  #34 0x0809e7a8 in make_child (s=0x80c9c80, slot=0) at
> >> prefork.c:680
> >> >> >  #35 0x0809ee53 in ap_mpm_run (_pconf=0xbf86d6f0,
> >> plog=0x81031a0,
> >> >> > s=0xbf86d6f4) at prefork.c:956
> >> >> >  #36 0x080622c8 in main (argc=2, argv=0xbf86d8b4) at main.c:717
> >> >> >  (gdb)
> >> >> >
> >> >> >
> >> >> >
> >> >> >
> >> >> > _______________________________________________
> >> >> > Mod_python mailing list
> >> >> > Mod_python at modpython.org
> >> >> > http://mailman.modpython.org/mailman/listinfo/mod_python
> >> >> >
> >> >> >
> >> >>
> >> >
> >>
> >>
>
>


More information about the Mod_python mailing list