|
Alec Matusis
matusis at yahoo.com
Tue Aug 5 04:55:41 EDT 2008
PS. Found Another way to get rid of the error: I had MaxRequestsPerChild 1 since this is my dev server and I wanted to be able to edit mod_python scripts without restarting apache. Increasing it to 10 does not fix it. Increasing to 100 makes it very rare. Increasing it to 20000 apparently fixes the problem (maybe I did not wait long enough) Perhaps what happens is that while the child is waiting on the MySQL connection, it gets EINTR event from the parent that is ready to die, because more requests came in, and so it dies before the child’s MySQL connection is complete? > -----Original Message----- > From: Alec Matusis [mailto:matusis at yahoo.com] > Sent: Tuesday, August 05, 2008 1:25 AM > To: 'Buck Golemon' > Cc: 'mod_python at modpython.org' > Subject: RE: [mod_python] intermittent bug with MySQLdb > > One more interesting piece of evidence: when I connect to the DB on the > same physical machine were my web scripts are, I can reproduce this > only when I connect to 127.0.0.1. When I connect to localhost, I do not > get the errors at all. This is because > > http://dev.mysql.com/doc/refman/5.0/en/connecting.html > > “For connections to localhost, MySQL programs attempt to connect to the > local server by using a Unix socket file. This occurs even if a --port > or -P option is given to specify a port number. To ensure that the > client makes a TCP/IP connection to the local server, use --host or -h > to specify a hostname value of 127.0.0.1” > > This last thing makes me think it’s a networking problem as you > suggest ? > > > From: Buck Golemon [mailto:workitharder at gmail.com] > Sent: Sunday, August 03, 2008 9:24 PM > To: Alec Matusis > Subject: Re: [mod_python] intermittent bug with MySQLdb > > Have you considered that this may be just a general problem of > networking / MySQL? That's my conclusion. I've gotten similar errors in > my high-concurrency applications (non mod-python) and to solve the > problem, wrote some code to re-try queryies on that error (among > others) and wrapped it around all my connections. > > As a counter-argument (if you want one), try doing a similar stress > test using a completely different client (maybe perl?) and show that > the error does not occur under similar or higher loads. > > --Buck > 2008/8/1 Alec Matusis <matusis at yahoo.com> > I was stress- testing MySQL replication on my development setup and run > into a particularly nasty case of a hard-to-reproduced bug that I have > encountered earlier. > > Under high concurrency, I sometimes get > > build/bdist.linux-i686/egg/MySQLdb/connections.py: > OperationalError: (2003, "Can't connect to MySQL server on ''10.18.0.2 > (4)") > > > Here (4) stangs for: > > #perror 4 > OS error code 4: Interrupted system call > > I am using apache 2.2.6 with worker MPM , python 2.4.4 and mod_python > 3.3.1. > > 10.18.0.2 is a remote DB test server. > I upgraded MySQLdb adapter from 1.2.0 to 1.2.2 and got the same error > with both versions. The adapters are compiled with > mysqlclient_r , which is supposed to be thread-safe. > > Then, I pointed MySQL connections to localhost, which runs an > identical copy of the same database. > > To get errors on localhost, I had to increase the request rate by about > 10x (no errors at all otherwise), and I got > OperationalError: (2003, "Can't connect to MySQL server on '127.0.0.1' > (4)") > 127.0.0.1 is my development machine, and I have never seen these errors > before, until I ran this stress test today, bringing the machine to > load average 8.0. > > 127.0.0.1 and 10.18.0.2 are two different machines, with different > kernels and even different MySQL versions, so it is the client machine > that is a problem. > > I have written a single-threaded stand-alone python loop, that simply > connects to the DB on the remote server 10.18.0.2 and executes "SELECT > 1". I have run several of these loops concurrently each doing about > 10000 iterations, and they never gave a connection error. > > MySQLdb README says: > MySQLdb is an interface to the popular MySQL_ database server > for > Python. The design goals are > - Thread-safety > - Thread-friendliness (threads will not block each other) > > So my guess is that this is a thread-safety problem having to do with > mod_python. > > I now recall that I have seen the problem earlier: > I have several identically configured production web servers connected > to a single DB server. I have seen these errors infrequently on 32bit > webservers, but extremely rarely or almost never on 64bit webservers. > I since phased out 32bit web servers to serve static files only, so I > do not have this data anymore. > > _______________________________________________ > Mod_python mailing list > Mod_python at modpython.org > http://mailman.modpython.org/mailman/listinfo/mod_python > > > > -- > Buck Golemon > AIM: bukzor > Cell: (480) 272-1153
|