Graham Swallow
gps at trix.dircon.co.uk
Wed Nov 15 19:56:52 EST 2000
On Tue, 14 Nov 2000, Alexis Iglauer wrote: > My datafiles are time series data, usually for 24 > hours at a time, so I will have a whole bunch of > datafiles which make up a contiguous set of data over > several days/months/years. I have a similar situation, but not in apache_mod. The data is an array of floats (binary portable), one every minute for a week, over the year. Repackaging last weeks data is slow, but you dont do that for every request. I use Array to get it as a string, then I use zlib to get a string, then I 'escape' the data for MySql (using the standard function) and write it into a blob field. The data comes back with no need for un-escaping (its so that it goes into an SQL statement). Then I call a C function to convert the floats to doubles and create a BLT/Vector. Its very quick. Using GD (or whatever) should not be slow, but a bit of C at the right place means you move chunks of data around, not iterate python over thousands of items. The 'most recent' file obviously needs special treatment. You could re-process today every hour (hint: move, start new file, then process the moved file, if fail - dont delete). That still leaves you with a 'double' data source, first SQL, then the additional file. Alternatively, get the data collector to post the data ... Graham
|