|
Graham Dumpleton
grahamd at dscpl.com.au
Tue Jan 23 23:53:48 EST 2007
Suresh Lakshmanan wrote ..
> Hi Graham,
> I'm not very familiar with the apache MPM model. Thanks for your references,
> im able to get an idea of how the subinterpreters work in mod_python.
>
> My python application is a django based application. It uses a few python
> modules like
> PIL (python imaging library), dnspython, django, libxx (our c++ extension
> written in python)
>
> Multiple requests to my django application (running inside apache with
> mod_python) are serialized and they execute one by one.
> Lot of them get 503 responses.
>
> Here's my apache conf.
Does running:
httpd -l
actually show worker.c in the list of files. Ie. to confirm that worker MPM is
actually compiled in.
> # worker MPM
> # StartServers: initial number of server processes to start
> # MaxClients: maximum number of simultaneous client connections
> # MinSpareThreads: minimum number of worker threads which are kept spare
> # MaxSpareThreads: maximum number of worker threads which are kept spare
> # ThreadsPerChild: constant number of worker threads in each server process
> # MaxRequestsPerChild: maximum number of requests a server process serves
> <IfModule worker.c>
> StartServers 1
> MaxClients 64
> MinSpareThreads 64
> MaxSpareThreads 64
> ThreadsPerChild 64
> MaxRequestsPerChild 0
> ServerLimit 1
This is what I was talking about with limiting the maximum number of child
process to 1. Because you have this, if a single request takes some time to
execute and the point it is waiting or taking time in is in C/C++ code in a
external Python module, and that module doesn't properly release the Python
GIL, all other requests will bank up behind that one request waiting for it to
return out of the C/C++ code.
In your libxx module, do you perform things in the C++ code that take a while
or do you make calls to stuff which can block. Do you ensure that you wrap time
consuming operations of code that can block in:
Py_BEGIN_ALLOW_THREADS
... blocking or time consuming operation.
Py_END_ALLOW_THREADS
> </IfModule>
>
> <Location "/reg/">
> SetEnv PYTHONHOME "/usr/local/bin/python2.4"
This will not actually do anything for mod_python itself. Would only affect CGI
scripts which are written in Python.
> SetHandler python-program
> PythonPath "['/usr/local/pi/applications/',
> '/usr/local/pi/nodes/data/regist
> ration', '/usr/local/pi/pimgr/lib', '/usr/local/pi/pimgr/lib/ca' ,
> '/usr/local/p
> i/pimgr/lib/MySQLdb'] + sys.path"
> PythonHandler django.core.handlers.modpython
> SetEnv PINET_HOME
> "/usr/local/pi/nodes/data/registration/pimgr/reg/ca/config
> "
> SetEnv PIMGR_PATH "/usr/local/pi/nodes/data/registration/pimgr"
> SetEnv DJANGO_SETTINGS_MODULE pimgr.settings
> PythonDebug On
> </Location>
>
> On 1/24/07, Graham Dumpleton <grahamd at dscpl.com.au> wrote:
> >
> > Suresh Lakshmanan wrote ..
> > > Hi,
> > > Is it possible for mod_python to create new subinterpreter when requests
> > > arrive? I was thinking that mod_python has a configurable pool of python
> > > interpreters
> > > and it will delegate a request to an interpreter from the pool.
> > >
> > > But, it looks like this is not the case.
> >
> > Correct, that is not the case. You are misunderstand what the
> > subinterpreters
> > are there for.
> >
> > > Only one subinterpreter is only
> > > created for a virtual host or directory config according to
> > documentation.
> >
> > More or less, yes. Each sub interpreter is named and is tied to some
> part
> > of
> > the URL namespace. The default is that there will be a sub interpreter
> for
> > each virtual host, but this can be overridden by various directives.
> >
> > > This is
> > > creating problems because the other requests have to wait for a long
> > time
> > > and i get 503 from apache.
> >
> > The way Python sub interpreters are used, it is highly unlikely that
> a 503
> > response is going to be because of them. It may be how you have Apache
> > configured, but to know you will have to state what platform you are
> using
> > and if on UNIX whether you have compiled Apache with the prefork MPM
> > or worker MPM. You also should post what you Apache configuration is
> > that controls the number of Apache sub processes and/or threads within
> > a process.
> >
> > > It will be useful if i have multiple
> > > subinterpreters
> > > to which apache can assign requests arbitrarily. I do not maintain
> state
> > > in
> > > any of the interpreters.
> >
> > This will not make any difference, what matters is how many Apache child
> > process are allowed to be created when using prefork MPM, or how many
> > child processes and/or threads with a process that Apache is allowed
> to
> > create when using a multithreaded MPM.
> >
> > If using prefork MPM and you have restricted the number of Apache child
> > processes to a small fixed number, then if you get more requests than
> > that arrive, there will be no processes available to handle the request.
> > If
> > you are using a multithreaded MPM, then a small number of threads across
> > all processes will limit how many requests can be handled concurrently.
> >
> > Creating extra Python subinterpreters only gives you separation of data,
> > it doesn't provide any additional concurrency abilities. You seem not
> to
> > understand this point.
> >
> > > iow, how do i scale this to a large number of users. how do i introduce
> > > parallel processing of requests for the same virtual host/directory.
> at
> > > the
> > > moment it looks
> > > like its single threaded.
> >
> > If you use prefork, then only one request in each child process can be
> > handled at a time, but then you should be running with adequate child
> > processes so that isn't a problem. If using a multithreaded MPM, then
> > there
> > already can be multiple requests handled at the same time and these
> > can be handled even within the same Python subinterpreter.
> >
> > > definitely there should be a way as this is a common problem... can
> > someone
> > > please point me to the solution.
> >
> > You probably need to research more about the different MPM models for
> > Apache. For starters, read:
> >
> >
> > http://www.dscpl.com.au/wiki/ModPython/Articles/TheProcessInterpreterModel
> >
> > You might then see what you can find in documentation about Apache
> > referenced on:
> >
> > http://wiki.apache.org/mod_python/
> >
> > On first guess I would say how you have configured Apache is limiting
> > things.
> >
> > If you are using a multithreaded MPM, then you might have a third party
> > module
> > for Python you are using which doesn't release the Python GIL around
> a
> > long
> > running operation being performed in C code. Thus, what extra Python
> > modules
> > are you using?
> >
> >
> > Graham
> >
|