|
Alec Matusis
matusis at yahoo.com
Wed Aug 6 01:43:57 EDT 2008
Sending it back to the list: I cannot get the errors with MaxRequestsPerChild 1 or MaxRequestsPerChild 20000 when I use prefork. Have you tried prefork? From: Buck Golemon [mailto:workitharder at gmail.com] Sent: Tuesday, August 05, 2008 6:11 PM To: Alec Matusis Subject: Re: [mod_python] intermittent bug with MySQLdb That's a pretty big "or". I was convinced it was MySQL networking until you posted this. Do you still have your +20000 setting above in effect? On 8/5/08, Alec Matusis <matusis at yahoo.com> wrote: I am now convinced that this is either a mod_python or MySQLdb threading bug. I recompiled apache with prefork instead of worker, and these MySQLdb errors disappeared. > -----Original Message----- > From: Alec Matusis [mailto:matusis at yahoo.com] > Sent: Tuesday, August 05, 2008 1:56 AM > To: 'Alec Matusis'; 'Buck Golemon' > Cc: mod_python at modpython.org > Subject: RE: [mod_python] intermittent bug with MySQLdb > > PS. > > Found Another way to get rid of the error: > > I had > MaxRequestsPerChild 1 > since this is my dev server and I wanted to be able to edit mod_python > scripts without restarting apache. > Increasing it to 10 does not fix it. > Increasing to 100 makes it very rare. > Increasing it to 20000 apparently fixes the problem (maybe I did not > wait > long enough) > > > Perhaps what happens is that while the child is waiting on the MySQL > connection, it gets EINTR event from the parent that is ready to die, > because more requests came in, and so it dies before the child's MySQL > connection is complete? > > > -----Original Message----- > > From: Alec Matusis [mailto:matusis at yahoo.com] > > Sent: Tuesday, August 05, 2008 1:25 AM > > To: 'Buck Golemon' > > Cc: 'mod_python at modpython.org' > > Subject: RE: [mod_python] intermittent bug with MySQLdb > > > > One more interesting piece of evidence: when I connect to the DB on > the > > same physical machine were my web scripts are, I can reproduce this > > only when I connect to 127.0.0.1. When I connect to localhost, I do > not > > get the errors at all. This is because > > > > http://dev.mysql.com/doc/refman/5.0/en/connecting.html > > > > "For connections to localhost, MySQL programs attempt to connect to > the > > local server by using a Unix socket file. This occurs even if a -- > port > > or -P option is given to specify a port number. To ensure that the > > client makes a TCP/IP connection to the local server, use --host or - > h > > to specify a hostname value of 127.0.0.1" > > > > This last thing makes me think it's a networking problem as you > > suggest ? > > > > > > From: Buck Golemon [mailto:workitharder at gmail.com] > > Sent: Sunday, August 03, 2008 9:24 PM > > To: Alec Matusis > > Subject: Re: [mod_python] intermittent bug with MySQLdb > > > > Have you considered that this may be just a general problem of > > networking / MySQL? That's my conclusion. I've gotten similar errors > in > > my high-concurrency applications (non mod-python) and to solve the > > problem, wrote some code to re-try queryies on that error (among > > others) and wrapped it around all my connections. > > > > As a counter-argument (if you want one), try doing a similar stress > > test using a completely different client (maybe perl?) and show that > > the error does not occur under similar or higher loads. > > > > --Buck > > 2008/8/1 Alec Matusis <matusis at yahoo.com> > > I was stress- testing MySQL replication on my development setup and > run > > into a particularly nasty case of a hard-to-reproduced bug that I > have > > encountered earlier. > > > > Under high concurrency, I sometimes get > > > > build/bdist.linux-i686/egg/MySQLdb/connections.py: > > OperationalError: (2003, "Can't connect to MySQL server on > ''10.18.0.2 > > (4)") > > > > > > Here (4) stangs for: > > > > #perror 4 > > OS error code 4: Interrupted system call > > > > I am using apache 2.2.6 with worker MPM , python 2.4.4 and mod_python > > 3.3.1. > > > > 10.18.0.2 is a remote DB test server. > > I upgraded MySQLdb adapter from 1.2.0 to 1.2.2 and got the same error > > with both versions. The adapters are compiled with > > mysqlclient_r , which is supposed to be thread-safe. > > > > Then, I pointed MySQL connections to localhost, which runs an > > identical copy of the same database. > > > > To get errors on localhost, I had to increase the request rate by > about > > 10x (no errors at all otherwise), and I got > > OperationalError: (2003, "Can't connect to MySQL server on > '127.0.0.1' > > (4)") > > 127.0.0.1 is my development machine, and I have never seen these > errors > > before, until I ran this stress test today, bringing the machine to > > load average 8.0. > > > > 127.0.0.1 and 10.18.0.2 are two different machines, with different > > kernels and even different MySQL versions, so it is the client > machine > > that is a problem. > > > > I have written a single-threaded stand-alone python loop, that simply > > connects to the DB on the remote server 10.18.0.2 and executes > "SELECT > > 1". I have run several of these loops concurrently each doing about > > 10000 iterations, and they never gave a connection error. > > > > MySQLdb README says: > > MySQLdb is an interface to the popular MySQL_ database server > > for > > Python. The design goals are > > - Thread-safety > > - Thread-friendliness (threads will not block each other) > > > > So my guess is that this is a thread-safety problem having to do with > > mod_python. > > > > I now recall that I have seen the problem earlier: > > I have several identically configured production web servers > connected > > to a single DB server. I have seen these errors infrequently on 32bit > > webservers, but extremely rarely or almost never on 64bit webservers. > > I since phased out 32bit web servers to serve static files only, so > I > > do not have this data anymore. > > > > _______________________________________________ > > Mod_python mailing list > > Mod_python at modpython.org > > http://mailman.modpython.org/mailman/listinfo/mod_python > > > > > > > > -- > > Buck Golemon > > AIM: bukzor > > Cell: (480) 272-1153 -- Buck Golemon AIM: bukzor Cell: (480) 272-1153
|