[mod_python] IPC, db conn sharing, and other fine things

>As far as the global dictionaries suggestion, that will work within any
>individual process, but each process will have to maintain their own
>disparate cache and registry. I'd thought about it, and I'll probably
>give it a shot just to see what the performance hit would be in running
>them in each process.


One simple way to solve this is to force Apache into threading. I'm using
the MPM worker here (Solaris), so that over 20 connections share the same
process (but different threads). This works well for publisher and common
handlers, but this will NOT work for the cgihandler (which only allows one
active thread). A big advantage of the (worker) threading mpm is that you
only need one Python interpreter 'in the air', saving a few megabytes for
each child.

Simple multi-read-single-write strategy:

cachelock = threading.Lock()
#...

try:
    result = cache[key]
except KeyError:
    cachelock.aquire()
    try:
        # Read again, since some other thread may have done
        # the same now
        try:
            result = cache[key]
        except KeyError:
            result = someFetchingFunction(key)
            cache[key] = result
    finally:
        cachelock.release()
return result

With multiple processes, each process will gather the neccesary cache data
as it goes, so initially the system will be a bit slower but ramps up rather
quick as more data is cached. However, this does eat quite some memory, all
cache data is duplicated for each process.

>For right now I've hacked together a flat file-based caching/registry
>system using libxml2 and fcntl locking. It's performance will work,
>though I really miss the speed of an all-in-memory caching/registry
>system. *sigh* You make do with what you can.

I think there are some Python shared memory (across processes) libraries,
they might be worth looking into. You could also write some caching code
based on shared memory in C and package it into a .so library and load it
into Python.

The filesystem is not so bad, most of it is cached in memory anyway. Some
OSses support in-memory files, that might be a nice one too.

Mike.



-- 
The information contained in this communication and any attachments is confidential and may be privileged, and is for the sole use of the intended recipient(s). Any unauthorized review, use, disclosure or distribution is prohibited. If you are not the intended recipient, please notify the sender immediately by replying to this message and destroy all copies of this message and any attachments. ASML is neither liable for the proper and complete transmission of the information contained in this communication, nor for any delay in its receipt.