[mod_python] need help solving some performance issues

On Thu, 10 Apr 2008, Colin Bean wrote:

> 
> Have you profiled your code at all?  Might help to track down problem areas...
> 

No I haven't.  Is there a way to profile it in the context of
mod_python?  A lot of what is happening is manipulating clearsilver datasets.

> 
> >  Currently my plan is to throw hardware at problem.  The traditional way to
> >  expand would be to add an additional webserver and do DNS round robin, or
> >  something like that.  Unfortunately this would cause database problems for
> >  me.  Top of my todo list is to swap out psycopg and use sqlrelay but I don't
> >  have enough time to do this quickly.  So... I am now considering switching to
> >  a dual cpu system.  I would keep apache in prefork mode-- I have read that
> >  this works fine on systems with 2 processors.
> >
> 
> Could you be more specific about the database problems you're
> anticipating?  Are you worried about the database server being able to
> scale as well?
> 

Right now I do not have connection pooling-- one process, one database
connection.  If I add a second web server I am going to run out of database
processes.  My postgresql server is on FreeBSD and increasing the available
processes for postgresql on FreeBSD is a major pain.

The database is doing fine-- hardly any load at all.  All the problems seem
to be in the application realm.  Connection pooling is definitely needed.

> 
> You might also want to look at caching solutions, especially if you
> have plenty of RAM :)  You could probably roll your own with Memcached
> without too much trouble, and this could potentially ease the load on
> the CPU quite a bit.

The database load doesn't seem to be really a big problem.  There are hundred
of thousands of pages, each translated into 6 languages.  I tinkered around
with mod_cache a bit but it couldn't handle it.

You are right-- trying to profile it is probably the first order of business.

Fyi the site in question is:

http://www.flashcardexchange.com/