[mod_python] Re: Django-mod_python taking time loading libraries

Graham Dumpleton graham.dumpleton at gmail.com
Sat Dec 26 05:15:21 EST 2009


The problem you are going to have using such an older version of
Django is its lack of thread safety. This means that you can only use
single threaded processes. When using mod_python that means Apache
prefork MPM. The problem with that is that every Apache server child
process will have a copy of every one of your Django sites. This will
use up a lot of memory. Using mod_wsgi embedded mode you would have
same issue.

What I would suggest therefore is instead create a mod_wsgi daemon
process group for each distinct application. You can then control the
number of single threaded processes that each site gets. For less
trafficked sites you might get away with two single threaded process,
for those with a bit more traffic somewhere between 5 and 10 depending
on load. Even those figures may be overkill so long as you properly
tune your application/database to remove bottlenecks.

For example, look at:

  http://collingrady.wordpress.com/2009/01/06/mod_python-versus-mod_wsgi/

This person was actually only using one single threaded processes and
still getting very good performance on a site which was getting a fair
bit of traffic. They had they optimised the hell out of their Django
application and database though and new what they were doing in that
area. In the end they may have switched to 2 or 3 processes, but that
was just to get rid of some startup request delays they were seeing
due to recycling processes after a set number of requests to eliminate
risk of memory creep if Python object references leaks, or spikes in
transient memory usage.

Anyway, split the sites out into separate daemon process groups. You
don't loose anything by doing this and gives you better control over
each as to how many resources you give them. Because they are separate
daemon process groups, also gives you ability to restart sites
individually without restarting whole of Apache. For what I mean by
this read:

  http://code.google.com/p/modwsgi/wiki/ReloadingSourceCode

Graham

2009/12/24 Graham Dumpleton <graham.dumpleton at gmail.com>:
> 2009/12/23 bharath venkatesh <bharathv6.project at gmail.com>:
>>>I note you have taken this off list.
>> sorry it is was not intended to take it off the list ,it was by mistake.
>>>How many Django applications?
>> we are using 4-5 Django application and we are using Django 0.97 (not sure
>> if this version is thread safe )
>
> No it isn't regarded as thread safe.
>
> I'll comment later when have time.
>
> Graham
>
>>>There are so many variables to all of this as far as knowing what is
>>>the best configuration and unfortunately only you know your
>>>applications so is going to be very hard for me to say that one way is
>>>going to be the best.
>> I understand ,we will do lot of testing and tuning before we arrive at the
>> final configuration .Thanks a lot for the help.
>> Thanks,
>> Bharath
>>
>>
>> On Wed, Dec 23, 2009 at 4:48 PM, Graham Dumpleton
>> <graham.dumpleton at gmail.com> wrote:
>>>
>>> I note you have taken this off list.
>>>
>>> 2009/12/23 bharath venkatesh <bharathv6.project at gmail.com>:
>>> > we are using apache for mainly mod python and django applications ,
>>> >  small
>>> > images such as logos are also served ( can be migrated ) and also one
>>> > mod_php application which receives very few request .
>>>
>>> How many Django applications?
>>>
>>> If many or memory requirements of each are large, then using embedded
>>> mode is a bad idea as only option is to have a copy of each in every
>>> process. So, if using one Apache for many Django instances, better off
>>> using daemon mode as gives better control over each application and
>>> can provision less processes for Django instances which are less used
>>> so overall memory is less. You can also make use of multithreading
>>> where appropriate, that is, can handle greater capacity requests where
>>> most of the time requests are stuck waiting on databases. Your code
>>> though then has to be thread safe.
>>>
>>> There are so many variables to all of this as far as knowing what is
>>> the best configuration and unfortunately only you know your
>>> applications so is going to be very hard for me to say that one way is
>>> going to be the best.
>>>
>>> Graham
>>>
>>> > On Wed, Dec 23, 2009 at 4:25 PM, Graham Dumpleton
>>> > <graham.dumpleton at gmail.com> wrote:
>>> >>
>>> >> What else is your Apache being used for? If it is being use for other
>>> >> stuff it may not be possible to tune Apache embedded in a way that is
>>> >> best for Python web application as that will be too detriment of other
>>> >> stuff you are hosting.
>>> >>
>>> >> So, are you hosting static media on same Apache or using a separate
>>> >> web server such as nginx for static media? Are you running PHP using
>>> >> mod_php on same Apache? Are you running anything else using any
>>> >> fastcgi/scgi/ajp hosting mechanisms?
>>> >>
>>> >> In short, unless that Apache is dedicated to just the Python web
>>> >> application, the trouble of trying to tune Apache for it is no worth
>>> >> the trouble and just better off using daemon mode thus allowing you to
>>> >> separately control things.
>>> >>
>>> >> BTW, depends a bit on what your site is doing, but using nginx proxy
>>> >> front end can be very advantageous as isolates Apache from slow
>>> >> clients allowing Apache processes/threads to be better utilised.
>>> >>
>>> >> Graham
>>> >>
>>> >> 2009/12/23 bharath venkatesh <bharathv6.project at gmail.com>:
>>> >> > Hi Graham,
>>> >> >
>>> >> >      Thanks for the link it was very informative
>>> >> > - Technically running embedded mode with prefork MPM should offer the
>>> >> > best
>>> >> > performance, especially for machines with many cpus/cores.
>>> >> > - just because mod_wsgi embedded mode and prefork MPM may be the
>>> >> > fastest
>>> >> > solution out there for Apache
>>> >> > As above lines (from link ) suggests ,can we use mod_wsgi embedded
>>> >> > mode
>>> >> > and
>>> >> > prefork MPM (as meantioned that it is best ) ?and tune apache not to
>>> >> > load
>>> >> > libraries and intialise every thime  (as our apache is already
>>> >> > running
>>> >> > as
>>> >> > prefork MPM and also our machines are multi core (8 cores) and has 16
>>> >> > AG
>>> >> > Ram)  or we have to use mod_wsgi in daemon mode to pervent huge
>>> >> > loading
>>> >> > and
>>> >> > intialising time ? but  as daemon mode is used with worker MPM we
>>> >> > have
>>> >> > to
>>> >> > recompile apache
>>> >> > Thanks,
>>> >> > Bharath
>>> >> >
>>> >> > On Wed, Dec 23, 2009 at 3:19 AM, Graham Dumpleton
>>> >> > <graham.dumpleton at gmail.com> wrote:
>>> >> >>
>>> >> >> 2009/12/23 bharath venkatesh <bharathv6.project at gmail.com>:
>>> >> >> > will mod wsgi in Daemon mode  solve this issue  as loading
>>> >> >> > libraries
>>> >> >> > and
>>> >> >> > initialising is done only once ?
>>> >> >>
>>> >> >> Maybe.
>>> >> >>
>>> >> >> As documented in:
>>> >> >>
>>> >> >>
>>> >> >>
>>> >> >>
>>> >> >>  http://blog.dscpl.com.au/2009/03/load-spikes-and-excessive-memory-usage.html
>>> >> >>
>>> >> >> the OP is having problems because of how embedded mode Python
>>> >> >> applications work in Apache. This may be  exacerbated by using
>>> >> >> prefork
>>> >> >> MPM with Apache. Same issues come up with mod_wsgi embedded mode as
>>> >> >> with mod_python if you do not configure your Apache correctly for
>>> >> >> fat
>>> >> >> Python web applications. If you are limited in being able to do that
>>> >> >> because Apache is heavily used for static file serving or mod_php,
>>> >> >> you
>>> >> >> really should use mod_wsgi daemon mode instead as described in that
>>> >> >> post.
>>> >> >>
>>> >> >> Why daemon mode will help is fixed number of persistent processes
>>> >> >> and
>>> >> >> more specifically defaulting to one process, so you only incur
>>> >> >> startup
>>> >> >> cost once and not for many processes, or in case of embedded mode
>>> >> >> also
>>> >> >> not when the processes get recycled as is default for Apache.
>>> >> >>
>>> >> >> Graham
>>> >> >>
>>> >> >> > On Tue, Dec 22, 2009 at 5:13 PM, bharath venkatesh
>>> >> >> > <bharathv6.project at gmail.com> wrote:
>>> >> >> >>
>>> >> >> >> Hi,
>>> >> >> >>
>>> >> >> >>     Django with mod python  is taking time loading libraries . I
>>> >> >> >> am
>>> >> >> >> using
>>> >> >> >> python  open source libraries like montylingua , simplejson and
>>> >> >> >> django
>>> >> >> >> is
>>> >> >> >> taking
>>> >> >> >> time loading these libraries .
>>> >> >> >>
>>> >> >> >> When django is run using python manage.py runserver first request
>>> >> >> >> takes
>>> >> >> >> time but rest of the requests doesn't take time ( quite fast )
>>> >> >> >> ,which
>>> >> >> >> is
>>> >> >> >> acceptable but when django is used along with apache and
>>> >> >> >> mod_python
>>> >> >> >> it
>>> >> >> >> takes
>>> >> >> >> time as much time as django takes for first request  when run
>>> >> >> >> using
>>> >> >> >> python
>>> >> >> >> manage.py runserver for almost every request.
>>> >> >> >>
>>> >> >> >> How to avoid this as our performance is greatly hindered by this
>>> >> >> >> behaviour
>>> >> >> >>
>>> >> >> >> is it possible to cache these external open source libraries  ?
>>> >> >> >>
>>> >> >> >> Thanks in Advance,
>>> >> >> >> Bharath
>>> >> >> >
>>> >> >> > _______________________________________________
>>> >> >> > Mod_python mailing list
>>> >> >> > Mod_python at modpython.org
>>> >> >> > http://mailman.modpython.org/mailman/listinfo/mod_python
>>> >> >> >
>>> >> >> >
>>> >> >
>>> >> >
>>> >
>>> >
>>
>>
>



More information about the Mod_python mailing list