[mod_python] IPC, db conn sharing, and other fine things

>- nsv buckets --- nsv == NaviServer (AOLServer's original name) shared
>variables... they were effectively arrays that stored information
>accessible and modifiable by any running thread... perfect for caching
>database information for faster generation of dynamic pages...


In mod_python, just use global variables in modules. Use dictionary objects
as fast storage mechanism.


>- database connection pooling --- provided easy, standardized, and
>scalable access to existing database connections


Same as above: Use global variables.

>... obviously I could write some sort of
>caching daemon in Python that used UNIX sockets and some simple
>key/value pair protocol to do db data caching... or I could write some
>sort of db-pooling daemon...

Why not integrate it into one of your modules?
Once loaded, the module remains in memory. So you can have it start a daemon
thread and use that as your IPC server. I always design the pages such that
no IPC is needed, each page gets its data exclusively from the request and
the database. The database is a great way to exchange data anyway, performs
excellent and survives even a power-down.

>So I suppose what I'm asking is that is there any way natively in
>mod_python to support these enterprise critical features? And if not,
>how the hell do you guys who use mod_python manage to have massive
>hit-per-second sites without data caching?

Python is so powerful a language, that creating these "enterprise critical
features" in raw code is just something you do on the side. I don't put
these in a library because there's more work in maintaining that library
than just entering the code. A threadsafe DB pooling system takes about 5
lines of Python code to program, and you need only one for the whole
project.

I do a little caching for 'static' tables, something like:

cache = {}
...
try:
    data = cache[key]
except KeyError:
    c = db.cursor()
    c.execute(somequery % key)
    data = c.fetchall()[0][0]
    cache[key] = data
return data


Also, I did some experiments with our  MySQL database, and found that the
overhead in a DB connection creation is very, very low when the DBMS is on
the same host. I was unable to measure any difference in performance,
compared to running an SQL statement, connecting to the socket just didn't
take any time. I implemented DB connection pooling anyway.

When tuning performance, I tend to optimize the design, SQL and index trees,
that helps performance a lot more than fiddling with fancy features.
Database pooling helped about 1%, changing a primary key from a,b to b,a
increased performance by 300%.

Mike.



-- 
The information contained in this communication and any attachments is confidential and may be privileged, and is for the sole use of the intended recipient(s). Any unauthorized review, use, disclosure or distribution is prohibited. If you are not the intended recipient, please notify the sender immediately by replying to this message and destroy all copies of this message and any attachments. ASML is neither liable for the proper and complete transmission of the information contained in this communication, nor for any delay in its receipt.