|
Graham Dumpleton
graham.dumpleton at gmail.com
Sat Dec 26 05:15:21 EST 2009
The problem you are going to have using such an older version of Django is its lack of thread safety. This means that you can only use single threaded processes. When using mod_python that means Apache prefork MPM. The problem with that is that every Apache server child process will have a copy of every one of your Django sites. This will use up a lot of memory. Using mod_wsgi embedded mode you would have same issue. What I would suggest therefore is instead create a mod_wsgi daemon process group for each distinct application. You can then control the number of single threaded processes that each site gets. For less trafficked sites you might get away with two single threaded process, for those with a bit more traffic somewhere between 5 and 10 depending on load. Even those figures may be overkill so long as you properly tune your application/database to remove bottlenecks. For example, look at: http://collingrady.wordpress.com/2009/01/06/mod_python-versus-mod_wsgi/ This person was actually only using one single threaded processes and still getting very good performance on a site which was getting a fair bit of traffic. They had they optimised the hell out of their Django application and database though and new what they were doing in that area. In the end they may have switched to 2 or 3 processes, but that was just to get rid of some startup request delays they were seeing due to recycling processes after a set number of requests to eliminate risk of memory creep if Python object references leaks, or spikes in transient memory usage. Anyway, split the sites out into separate daemon process groups. You don't loose anything by doing this and gives you better control over each as to how many resources you give them. Because they are separate daemon process groups, also gives you ability to restart sites individually without restarting whole of Apache. For what I mean by this read: http://code.google.com/p/modwsgi/wiki/ReloadingSourceCode Graham 2009/12/24 Graham Dumpleton <graham.dumpleton at gmail.com>: > 2009/12/23 bharath venkatesh <bharathv6.project at gmail.com>: >>>I note you have taken this off list. >> sorry it is was not intended to take it off the list ,it was by mistake. >>>How many Django applications? >> we are using 4-5 Django application and we are using Django 0.97 (not sure >> if this version is thread safe ) > > No it isn't regarded as thread safe. > > I'll comment later when have time. > > Graham > >>>There are so many variables to all of this as far as knowing what is >>>the best configuration and unfortunately only you know your >>>applications so is going to be very hard for me to say that one way is >>>going to be the best. >> I understand ,we will do lot of testing and tuning before we arrive at the >> final configuration .Thanks a lot for the help. >> Thanks, >> Bharath >> >> >> On Wed, Dec 23, 2009 at 4:48 PM, Graham Dumpleton >> <graham.dumpleton at gmail.com> wrote: >>> >>> I note you have taken this off list. >>> >>> 2009/12/23 bharath venkatesh <bharathv6.project at gmail.com>: >>> > we are using apache for mainly mod python and django applications , >>> > small >>> > images such as logos are also served ( can be migrated ) and also one >>> > mod_php application which receives very few request . >>> >>> How many Django applications? >>> >>> If many or memory requirements of each are large, then using embedded >>> mode is a bad idea as only option is to have a copy of each in every >>> process. So, if using one Apache for many Django instances, better off >>> using daemon mode as gives better control over each application and >>> can provision less processes for Django instances which are less used >>> so overall memory is less. You can also make use of multithreading >>> where appropriate, that is, can handle greater capacity requests where >>> most of the time requests are stuck waiting on databases. Your code >>> though then has to be thread safe. >>> >>> There are so many variables to all of this as far as knowing what is >>> the best configuration and unfortunately only you know your >>> applications so is going to be very hard for me to say that one way is >>> going to be the best. >>> >>> Graham >>> >>> > On Wed, Dec 23, 2009 at 4:25 PM, Graham Dumpleton >>> > <graham.dumpleton at gmail.com> wrote: >>> >> >>> >> What else is your Apache being used for? If it is being use for other >>> >> stuff it may not be possible to tune Apache embedded in a way that is >>> >> best for Python web application as that will be too detriment of other >>> >> stuff you are hosting. >>> >> >>> >> So, are you hosting static media on same Apache or using a separate >>> >> web server such as nginx for static media? Are you running PHP using >>> >> mod_php on same Apache? Are you running anything else using any >>> >> fastcgi/scgi/ajp hosting mechanisms? >>> >> >>> >> In short, unless that Apache is dedicated to just the Python web >>> >> application, the trouble of trying to tune Apache for it is no worth >>> >> the trouble and just better off using daemon mode thus allowing you to >>> >> separately control things. >>> >> >>> >> BTW, depends a bit on what your site is doing, but using nginx proxy >>> >> front end can be very advantageous as isolates Apache from slow >>> >> clients allowing Apache processes/threads to be better utilised. >>> >> >>> >> Graham >>> >> >>> >> 2009/12/23 bharath venkatesh <bharathv6.project at gmail.com>: >>> >> > Hi Graham, >>> >> > >>> >> > Thanks for the link it was very informative >>> >> > - Technically running embedded mode with prefork MPM should offer the >>> >> > best >>> >> > performance, especially for machines with many cpus/cores. >>> >> > - just because mod_wsgi embedded mode and prefork MPM may be the >>> >> > fastest >>> >> > solution out there for Apache >>> >> > As above lines (from link ) suggests ,can we use mod_wsgi embedded >>> >> > mode >>> >> > and >>> >> > prefork MPM (as meantioned that it is best ) ?and tune apache not to >>> >> > load >>> >> > libraries and intialise every thime (as our apache is already >>> >> > running >>> >> > as >>> >> > prefork MPM and also our machines are multi core (8 cores) and has 16 >>> >> > AG >>> >> > Ram) or we have to use mod_wsgi in daemon mode to pervent huge >>> >> > loading >>> >> > and >>> >> > intialising time ? but as daemon mode is used with worker MPM we >>> >> > have >>> >> > to >>> >> > recompile apache >>> >> > Thanks, >>> >> > Bharath >>> >> > >>> >> > On Wed, Dec 23, 2009 at 3:19 AM, Graham Dumpleton >>> >> > <graham.dumpleton at gmail.com> wrote: >>> >> >> >>> >> >> 2009/12/23 bharath venkatesh <bharathv6.project at gmail.com>: >>> >> >> > will mod wsgi in Daemon mode solve this issue as loading >>> >> >> > libraries >>> >> >> > and >>> >> >> > initialising is done only once ? >>> >> >> >>> >> >> Maybe. >>> >> >> >>> >> >> As documented in: >>> >> >> >>> >> >> >>> >> >> >>> >> >> >>> >> >> http://blog.dscpl.com.au/2009/03/load-spikes-and-excessive-memory-usage.html >>> >> >> >>> >> >> the OP is having problems because of how embedded mode Python >>> >> >> applications work in Apache. This may be exacerbated by using >>> >> >> prefork >>> >> >> MPM with Apache. Same issues come up with mod_wsgi embedded mode as >>> >> >> with mod_python if you do not configure your Apache correctly for >>> >> >> fat >>> >> >> Python web applications. If you are limited in being able to do that >>> >> >> because Apache is heavily used for static file serving or mod_php, >>> >> >> you >>> >> >> really should use mod_wsgi daemon mode instead as described in that >>> >> >> post. >>> >> >> >>> >> >> Why daemon mode will help is fixed number of persistent processes >>> >> >> and >>> >> >> more specifically defaulting to one process, so you only incur >>> >> >> startup >>> >> >> cost once and not for many processes, or in case of embedded mode >>> >> >> also >>> >> >> not when the processes get recycled as is default for Apache. >>> >> >> >>> >> >> Graham >>> >> >> >>> >> >> > On Tue, Dec 22, 2009 at 5:13 PM, bharath venkatesh >>> >> >> > <bharathv6.project at gmail.com> wrote: >>> >> >> >> >>> >> >> >> Hi, >>> >> >> >> >>> >> >> >> Django with mod python is taking time loading libraries . I >>> >> >> >> am >>> >> >> >> using >>> >> >> >> python open source libraries like montylingua , simplejson and >>> >> >> >> django >>> >> >> >> is >>> >> >> >> taking >>> >> >> >> time loading these libraries . >>> >> >> >> >>> >> >> >> When django is run using python manage.py runserver first request >>> >> >> >> takes >>> >> >> >> time but rest of the requests doesn't take time ( quite fast ) >>> >> >> >> ,which >>> >> >> >> is >>> >> >> >> acceptable but when django is used along with apache and >>> >> >> >> mod_python >>> >> >> >> it >>> >> >> >> takes >>> >> >> >> time as much time as django takes for first request when run >>> >> >> >> using >>> >> >> >> python >>> >> >> >> manage.py runserver for almost every request. >>> >> >> >> >>> >> >> >> How to avoid this as our performance is greatly hindered by this >>> >> >> >> behaviour >>> >> >> >> >>> >> >> >> is it possible to cache these external open source libraries ? >>> >> >> >> >>> >> >> >> Thanks in Advance, >>> >> >> >> Bharath >>> >> >> > >>> >> >> > _______________________________________________ >>> >> >> > Mod_python mailing list >>> >> >> > Mod_python at modpython.org >>> >> >> > http://mailman.modpython.org/mailman/listinfo/mod_python >>> >> >> > >>> >> >> > >>> >> > >>> >> > >>> > >>> > >> >> >
|