[mod_python] need help solving some performance issues

On Apr 10, 2008, at 2:53 PM, Culley Harrelson wrote:
>>
>> You might also want to look at caching solutions, especially if you
>> have plenty of RAM :)  You could probably roll your own with  
>> Memcached
>> without too much trouble, and this could potentially ease the load on
>> the CPU quite a bit.
>
> The database load doesn't seem to be really a big problem.  There  
> are hundred
> of thousands of pages, each translated into 6 languages.  I tinkered  
> around
> with mod_cache a bit but it couldn't handle it.
>

As a better way to tackle that.

We have a mod_perl based application ( that is being transfered to  
mod_python )
and during high load period we found with profiling that there was no  
area
that we could improve it was just the entire process was taking too  
long and
to many reuqests were backing up and eventually either two many apache
processes were launched and swapping and thrashing started or we had  
timeouts.

this on code that took far less than 1 second from start to end. the  
code in question
was mostly dynamically generating html, we spend about 60 minutes  
rewriting the code
to store the final output for each different dynamic request in  
memcache and configuring
out front end nginx server to do the following:

check for file
check in memcache
check with application

the 2nd step was the new part and we havent had a peep since then.  
server load ( debian linux )
hasnt crossed much beyond .5 since then and is usually below .3

as a bonus, our client side response time went down because the front  
end static server is competing
less with the dynamic server for cpu time.