|
Luke Burden
luke at demonware.net
Fri Nov 28 17:35:47 EST 2008
Hi Graham, I'm a developer for a internally facing web application, which uses apache + mod_python + mysql. As the application is being increasingly used, I've noticed several errors occurring sporadically, but regularly enough to be cause for concern. Concern morphed into alarm when I found I could reproduce the problems by quickly and repeatedly refreshing a page that causes a number of queries to be executed! The problem manifests itself in either bad behavior from the python MySQLDb module: InterfaceError: (0, '') OperationalError: (2013, 'Lost connection to MySQL server during query') ProgrammingError: (2014, "Commands out of sync; you can't run this command now") (<- might be unrelated) .. and NoneType result sets where the data is basically not there, when it should be. Or in apache seg-faulting: [Fri Nov 28 21:34:42 2008] [notice] mod_python: Creating 8 session mutexes based on 6 max processes and 25 max threads. [Fri Nov 28 21:34:42 2008] [notice] mod_python: using mutex_directory /tmp [Fri Nov 28 21:34:42 2008] [notice] Apache/2.2.4 (Ubuntu) mod_python/3.3.1 Python/2.5.1 configured -- resuming normal operations [Fri Nov 28 21:35:03 2008] [error] [client 192.168.179.1] File does not exist: /var/www/favicon.ico [Fri Nov 28 21:35:04 2008] [error] [client 192.168.179.1] File does not exist: /var/www/favicon.ico [Fri Nov 28 21:35:06 2008] [error] [client 192.168.179.1] File does not exist: /var/www/favicon.ico [Fri Nov 28 21:35:06 2008] [error] [client 192.168.179.1] File does not exist: /var/www/favicon.ico [Fri Nov 28 21:35:07 2008] [notice] child pid 17523 exit signal Segmentation fault (11) [Fri Nov 28 21:35:07 2008] [error] [client 192.168.179.1] File does not exist: /var/www/favicon.ico [Fri Nov 28 21:35:07 2008] [error] [client 192.168.179.1] File does not exist: /var/www/favicon.ico [Fri Nov 28 21:35:08 2008] [error] [client 192.168.179.1] File does not exist: /var/www/favicon.ico [Fri Nov 28 21:35:15 2008] [notice] child pid 17525 exit signal Segmentation fault (11) [Fri Nov 28 21:35:15 2008] [notice] child pid 17579 exit signal Segmentation fault (11) [Fri Nov 28 21:35:16 2008] [error] [client 192.168.179.1] File does not exist: /var/www/favicon.ico [Fri Nov 28 21:35:17 2008] [notice] child pid 17607 exit signal Segmentation fault (11) [Fri Nov 28 21:35:19 2008] [error] [client 192.168.179.1] File does not exist: /var/www/favicon.ico ... etc ... A closer look at one of the segmentation faults: *** glibc detected *** /usr/sbin/apache2: double free or corruption (!prev): 0x000000000212a1c0 *** ======= Backtrace: ========= /lib/libc.so.6[0x2b6915ed7b0a] /lib/libc.so.6(cfree+0x8c)[0x2b6915edb6fc] /usr/lib/libmysqlclient_r.so.15(vio_delete+0x19)[0x2aaaadef6ff9] /usr/lib/libmysqlclient_r.so.15(end_server+0x2f)[0x2aaaadef2eff] /usr/lib/libmysqlclient_r.so.15(cli_safe_read+0x2b)[0x2aaaadef305b] /usr/lib/libmysqlclient_r.so.15(cli_advanced_command+0x195)[0x2aaaadef5675] /usr/lib/libmysqlclient_r.so.15(mysql_ping+0x31)[0x2aaaadec71a1] /var/lib/python-support/python2.5/_mysql.so[0x2aaaadc5e7e6] /usr/lib/libpython2.5.so.1.0(PyEval_EvalFrameEx+0x6335)[0x2b691a2ccd15] /usr/lib/libpython2.5.so.1.0(PyEval_EvalCodeEx+0x836)[0x2b691a2cdb06] /usr/lib/libpython2.5.so.1.0(PyEval_EvalFrameEx+0x553d)[0x2b691a2cbf1d] /usr/lib/libpython2.5.so.1.0(PyEval_EvalFrameEx+0x6564)[0x2b691a2ccf44] /usr/lib/libpython2.5.so.1.0(PyEval_EvalFrameEx+0x6564)[0x2b691a2ccf44] /usr/lib/libpython2.5.so.1.0(PyEval_EvalCodeEx+0x836)[0x2b691a2cdb06] /usr/lib/libpython2.5.so.1.0(PyEval_EvalFrameEx+0x553d)[0x2b691a2cbf1d] /usr/lib/libpython2.5.so.1.0(PyEval_EvalCodeEx+0x836)[0x2b691a2cdb06] /usr/lib/libpython2.5.so.1.0(PyEval_EvalFrameEx+0x553d)[0x2b691a2cbf1d] /usr/lib/libpython2.5.so.1.0(PyEval_EvalFrameEx+0x6564)[0x2b691a2ccf44] /usr/lib/libpython2.5.so.1.0(PyEval_EvalCodeEx+0x836)[0x2b691a2cdb06] /usr/lib/libpython2.5.so.1.0[0x2b691a26dc5e] /usr/lib/libpython2.5.so.1.0(PyObject_Call+0x13)[0x2b691a24c323] /usr/lib/libpython2.5.so.1.0[0x2b691a253aad] /usr/lib/libpython2.5.so.1.0(PyObject_Call+0x13)[0x2b691a24c323] /usr/lib/libpython2.5.so.1.0[0x2b691a24c547] /usr/lib/libpython2.5.so.1.0(PyObject_CallMethod+0xc0)[0x2b691a24e610] /usr/lib/apache2/modules/mod_python.so[0x2b6919ff91bc] /usr/lib/apache2/modules/mod_python.so[0x2b6919ffb687] /usr/sbin/apache2(ap_run_handler+0x7a)[0x437c4a] /usr/sbin/apache2(ap_invoke_handler+0x7c)[0x43affc] /usr/sbin/apache2(ap_process_request+0x178)[0x447248] /usr/sbin/apache2[0x44468c] /usr/sbin/apache2(ap_run_process_connection+0x72)[0x43ec22] /usr/sbin/apache2[0x44b696] /lib/libpthread.so.0[0x2b6915a4d317] /lib/libc.so.6(clone+0x6d)[0x2b6915f39d5d] ======= Memory map: ======== [Fri Nov 28 02:52:57 2008] [notice] child pid 28387 exit signal Aborted (6) Seems to clearly point the finger at the mysql client library. Google shows a few similar looking cases, usually solved by removing PHP, which in this case is not installed. Nor is expat XML parser used. Nor mhash.so. :) The problem occurs on both Ubuntu 7.10, and 8.04 installs using the default packages for apache, python, mod python, mysql. From the 7.10 instance: luke at luke-ubuntu:~$ dpkg -l | grep mysql ii libdbd-mysql-perl 4.004-2 A Perl5 database interface to the MySQL data ii libmysqlclient15off 5.0.45-1ubuntu3.3 MySQL database client library ii mysql-client-5.0 5.0.45-1ubuntu3.3 MySQL database client binaries ii mysql-common 5.0.45-1ubuntu3.3 MySQL database common files ii mysql-server 5.0.45-1ubuntu3.3 MySQL database server (meta package dependin ii mysql-server-5.0 5.0.45-1ubuntu3.3 MySQL database server binaries ii python-mysqldb 1.2.2-3ubuntu1 A Python interface to MySQL luke at luke-ubuntu:~$ dpkg -l | grep mod-python ii libapache2-mod-python 3.3.1-2 Apache 2 module that embeds Python within th All running on Python 2.5. I'm thinking of getting the source and building the libmysqlclient myself now, but I'm hoping you've got some wise suggestions for me. Thanks very much for your time! Regards, Luke
|