[mod_python] how to run multiple subinterpreters from mod_python

Suresh Lakshmanan suresh.lakshmanan at gmail.com
Tue Jan 23 23:09:14 EST 2007

Hi Graham,
I'm not very familiar with the apache MPM model. Thanks for your references,
im able to get an idea of how the subinterpreters work in mod_python.

My python application is a django based application. It uses a few python
modules like
PIL (python imaging library), dnspython, django, libxx (our c++ extension
written in python)

Multiple requests to my django application (running inside apache with
mod_python) are serialized and they execute one by one.
Lot of them get 503 responses.

Here's my apache conf.
# worker MPM
# StartServers: initial number of server processes to start
# MaxClients: maximum number of simultaneous client connections
# MinSpareThreads: minimum number of worker threads which are kept spare
# MaxSpareThreads: maximum number of worker threads which are kept spare
# ThreadsPerChild: constant number of worker threads in each server process
# MaxRequestsPerChild: maximum number of requests a server process serves
<IfModule worker.c>
StartServers         1
MaxClients          64
MinSpareThreads     64
MaxSpareThreads     64
ThreadsPerChild     64
MaxRequestsPerChild  0
ServerLimit          1

<Location "/reg/">
    SetEnv PYTHONHOME "/usr/local/bin/python2.4"
    SetHandler python-program
    PythonPath "['/usr/local/pi/applications/',
ration', '/usr/local/pi/pimgr/lib', '/usr/local/pi/pimgr/lib/ca' ,
i/pimgr/lib/MySQLdb'] + sys.path"
    PythonHandler django.core.handlers.modpython
    SetEnv PIMGR_PATH "/usr/local/pi/nodes/data/registration/pimgr"
    SetEnv DJANGO_SETTINGS_MODULE pimgr.settings
    PythonDebug On

On 1/24/07, Graham Dumpleton <grahamd at dscpl.com.au> wrote:
> Suresh Lakshmanan wrote ..
> > Hi,
> > Is it possible for mod_python to create new subinterpreter when requests
> > arrive? I was thinking that mod_python has a configurable pool of python
> > interpreters
> > and it will delegate a request to an interpreter from the pool.
> >
> > But, it looks like this is not the case.
> Correct, that is not the case. You are misunderstand what the
> subinterpreters
> are there for.
> > Only one subinterpreter is only
> > created for a virtual host or directory config according to
> documentation.
> More or less, yes. Each sub interpreter is named and is tied to some part
> of
> the URL namespace. The default is that there will be a sub interpreter for
> each virtual host, but this can be overridden by various directives.
> > This is
> > creating problems because the other requests have to wait for a long
> time
> > and i get 503 from apache.
> The way Python sub interpreters are used, it is highly unlikely that a 503
> response is going to be because of them. It may be how you have Apache
> configured, but to know you will have to state what platform you are using
> and if on UNIX whether you have compiled Apache with the prefork MPM
> or worker MPM. You also should post what you Apache configuration is
> that controls the number of Apache sub processes and/or threads within
> a process.
> > It will be useful if i have multiple
> > subinterpreters
> > to which apache can assign requests arbitrarily. I do not maintain state
> > in
> > any of the interpreters.
> This will not make any difference, what matters is how many Apache child
> process are allowed to be created when using prefork MPM, or how many
> child processes and/or threads with a process that Apache is allowed to
> create when using a multithreaded MPM.
> If using prefork MPM and you have restricted the number of Apache child
> processes to a small fixed number, then if you get more requests than
> that arrive, there will be no processes available to handle the request.
> If
> you are using a multithreaded MPM, then a small number of threads across
> all processes will limit how many requests can be handled concurrently.
> Creating extra Python subinterpreters only gives you separation of data,
> it doesn't provide any additional concurrency abilities. You seem not to
> understand this point.
> > iow, how do i scale this to a large number of users. how do i introduce
> > parallel processing of requests for the same virtual host/directory. at
> > the
> > moment it looks
> > like its single threaded.
> If you use prefork, then only one request in each child process can be
> handled at a time, but then you should be running with adequate child
> processes so that isn't a problem. If using a multithreaded MPM, then
> there
> already can be multiple requests handled at the same time and these
> can be handled even within the same Python subinterpreter.
> > definitely there should be a way as this is a common problem... can
> someone
> > please point me to the solution.
> You probably need to research more about the different MPM models for
> Apache. For starters, read:
> http://www.dscpl.com.au/wiki/ModPython/Articles/TheProcessInterpreterModel
> You might then see what you can find in documentation about Apache
> referenced on:
>   http://wiki.apache.org/mod_python/
> On first guess I would say how you have configured Apache is limiting
> things.
> If you are using a multithreaded MPM, then you might have a third party
> module
> for Python you are using which doesn't release the Python GIL around a
> long
> running operation being performed in C code. Thus, what extra Python
> modules
> are you using?
> Graham
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mm_cfg_has_not_been_edited_to_set_host_domains/pipermail/mod_python/attachments/20070124/f5b92723/attachment.html

More information about the Mod_python mailing list