mike.looijmans at asml.com
Thu Aug 28 08:29:58 EST 2003
>As far as the global dictionaries suggestion, that will work within any >individual process, but each process will have to maintain their own >disparate cache and registry. I'd thought about it, and I'll probably >give it a shot just to see what the performance hit would be in running >them in each process. One simple way to solve this is to force Apache into threading. I'm using the MPM worker here (Solaris), so that over 20 connections share the same process (but different threads). This works well for publisher and common handlers, but this will NOT work for the cgihandler (which only allows one active thread). A big advantage of the (worker) threading mpm is that you only need one Python interpreter 'in the air', saving a few megabytes for each child. Simple multi-read-single-write strategy: cachelock = threading.Lock() #... try: result = cache[key] except KeyError: cachelock.aquire() try: # Read again, since some other thread may have done # the same now try: result = cache[key] except KeyError: result = someFetchingFunction(key) cache[key] = result finally: cachelock.release() return result With multiple processes, each process will gather the neccesary cache data as it goes, so initially the system will be a bit slower but ramps up rather quick as more data is cached. However, this does eat quite some memory, all cache data is duplicated for each process. >For right now I've hacked together a flat file-based caching/registry >system using libxml2 and fcntl locking. It's performance will work, >though I really miss the speed of an all-in-memory caching/registry >system. *sigh* You make do with what you can. I think there are some Python shared memory (across processes) libraries, they might be worth looking into. You could also write some caching code based on shared memory in C and package it into a .so library and load it into Python. The filesystem is not so bad, most of it is cached in memory anyway. Some OSses support in-memory files, that might be a nice one too. Mike. -- The information contained in this communication and any attachments is confidential and may be privileged, and is for the sole use of the intended recipient(s). Any unauthorized review, use, disclosure or distribution is prohibited. If you are not the intended recipient, please notify the sender immediately by replying to this message and destroy all copies of this message and any attachments. ASML is neither liable for the proper and complete transmission of the information contained in this communication, nor for any delay in its receipt.