[mod_python] need help solving some performance issues

On Thu, Apr 10, 2008 at 9:57 AM, Culley Harrelson <harrelson at gmail.com> wrote:
> I manage a mod_python application that was built about 4 years ago and has
>  not changed much since.  It uses clearsilver templates and interacts with
>  postgresql via psychopg.  The database is on a second machine and the
>  mod_python machine (FreeBSD) has no other major applications.
>
>  The application is much more popular than it was when it was built and I am
>  now encountering some performance problems.  Specifically the CPU seems to be
>  the bottleneck.  The machine has 2 gigs of ram and a single processor.  Ram
>  doesn't seem to be a problem.  When examining top output, I see that every
>  once in a while a process comes along and sucks down 10%-20% of the cpu.  A
>  handful of these processes can (and regularly do) consume the whole CPU.
>
>  I don't really know how to go about optimizing the CPU usage of my
>  application.  Any suggestions are extremely welcome.
>


Have you profiled your code at all?  Might help to track down problem areas...


>  Currently my plan is to throw hardware at problem.  The traditional way to
>  expand would be to add an additional webserver and do DNS round robin, or
>  something like that.  Unfortunately this would cause database problems for
>  me.  Top of my todo list is to swap out psycopg and use sqlrelay but I don't
>  have enough time to do this quickly.  So... I am now considering switching to
>  a dual cpu system.  I would keep apache in prefork mode-- I have read that
>  this works fine on systems with 2 processors.
>

Could you be more specific about the database problems you're
anticipating?  Are you worried about the database server being able to
scale as well?


You might also want to look at caching solutions, especially if you
have plenty of RAM :)  You could probably roll your own with Memcached
without too much trouble, and this could potentially ease the load on
the CPU quite a bit.

Colin