[mod_python] high memory consumption with multiple Python interpreters ?

On 11/16/06, Richard Lewis <richardlewis at fastmail.co.uk> wrote:
> On Thursday 16 November 2006 12:47, Julien Cigar wrote:
> > (how "heavy" is a Python interpreter ?) ?
> >
> Assuming you're on Unix, I would suggest that you try it out. Try loading some
> python processing and importing the modules that you use. Then drop back into
> the shell and go:
>
> repeat:
> $ python -ic "import my_module" &
> several times
>
> $ ps aux | head -1
> USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
> $ ps aux | grep python
> r  7884  0.5  1.6  11964  8576 pts/4    T    13:00   0:00 python -ic import m
> r  7885  0.4  1.6  11972  8576 pts/4    T    13:00   0:00 python -ic import m
> r  7886  0.7  1.6  11964  8576 pts/4    T    13:00   0:00 python -ic import m
> r  7887  0.5  1.6  11972  8580 pts/4    T    13:00   0:00 python -ic import m
> r  7888  0.6  1.6  11964  8576 pts/4    T    13:00   0:00 python -ic import m
> r  7889  0.5  1.6  11964  8576 pts/4    T    13:00   0:00 python -ic import m
> r  7892  1.5  1.6  11964  8576 pts/4    T    13:00   0:00 python -ic import m
> r  7894  0.0  0.1   1952   632 pts/4    S+   13:00   0:00 grep python
>
> As you can see, my python processes are each using 1.6% of my computer's 512MB
> of RAM. Its not the most accurate test, but its quite fun ;-)

You also need to realize that having one process run multiple instances
of a Python interpreter (as with mod_python) is different than running
multiple processes entirely.  There are a lot of resources that will be
shared in the former case (such as all the C-based code that makes up
the Python core as well as any C-based python modules that are loaded).

The real memory usage is also highly dependent upon what your
mod_python handlers are doing.  If they are creating lots of objects,
store internal caches, load entire files into memory, do select * from
large database tables, etc.

Of course the only real way to tell what happens is to just try it out under
an actual Apache environment.  But aside from any mod_python memory
leaks (which tend to get fixed rather quickly when found), I think you'll
find that using multiple interpreters under mod_python is actually less
memory intensive (or no worse) than a multiple-process approach like
CGI.
-- 
Deron Meranda