MySQL Bugs: #35291: InnoDB crashes with Mutex at 0xb79c20b4 created file log0log.c line 744, lock va

Bug #35291	InnoDB crashes with Mutex at 0xb79c20b4 created file log0log.c line 744, lock va
Submitted:	14 Mar 2008 12:55	Modified:	9 Jul 2008 13:51
Reporter:	Glenn Matthys	Email Updates:
Status:	No Feedback	Impact on me:	None
Category:	MySQL Server: InnoDB storage engine	Severity:	S1 (Critical)
Version:	mysql Ver 14.12 Distrib 5.0.41, for pc-	OS:	Linux (Linux mysql1 2.6.21-ARCH #1 SMP PREEMPT)
Assigned to:	Assigned Account	CPU Architecture:	Any

Description:
Hello, 

This morning during the backups, using the command: 
nice -n10 mysqldump -u root -p"xxx" --all-databases 

(datadir is set to /home/mysql) The MySQL server crashed with the following message in /home/mysql/mysql1.err: 

InnoDB: Warning: a long semaphore wait: 
--Thread 2960063376 has waited at ./../include/log0log.ic line 315 for 740.00 seconds the semaphore: 
Mutex at 0xb79c20b4 created file log0log.c line 744, lock var 0 
waiters flag 0 
wait has ended 
InnoDB: Warning: a long semaphore wait: 
--Thread 2966961040 has waited at ./../include/log0log.ic line 315 for 740.00 seconds the semaphore: 
Mutex at 0xb79c20b4 created file log0log.c line 744, lock var 1 
waiters flag 0 
InnoDB: Warning: a long semaphore wait: 

After this the database server didn't accept new connections with the message " Too many connections" or " Can't connect to MySQL server on 'mysql1'". 

It's a small database server, it only has 128M of data. 

At that time another cronjob was running, performing alot of SELECTs, a few INSERTs and alot of UPDATE statements.

The server had been running for quite a long time with this combo without problems. The server has two CPUs, SMP is enabled.

mysql --version
mysql  Ver 14.12 Distrib 5.0.41, for pc-linux-gnu (i686) using readline 5.2

How to repeat:
At this time I haven't been to recreate this issue. It's probably an SMP race condition that happens every once in a blue moon.

Glenn,

please post the entire .err log, gzipped.

--Heikki

This is the entire err log:
http://users.opengate.be/~glenn/mysqlcrash/mysql1.err.gz

Version: '5.0.41-log'  socket: '/tmp/mysql.sock'  port: 3306  Source distribution
InnoDB: Warning: a long semaphore wait:
--Thread 2960063376 has waited at ./../include/log0log.ic line 315 for 740.00 seconds the semaphore:
Mutex at 0xb79c20b4 created file log0log.c line 744, lock var 0
waiters flag 0
wait has ended
InnoDB: Warning: a long semaphore wait:
--Thread 2966961040 has waited at ./../include/log0log.ic line 315 for 740.00 seconds the semaphore:
Mutex at 0xb79c20b4 created file log0log.c line 744, lock var 1
waiters flag 0
InnoDB: Warning: a long semaphore wait:
080314  7:47:06 [Note] /usr/sbin/mysqld: Normal shutdown

--Thread 2945895312 has waited at log0log.c line 204 for 12833.00 seconds the semaphore:
Mutex at 0xb79c20b4 created file log0log.c line 744, lock var 0
waiters flag 0
wait has ended

Hmm... the threads are waiting at the log mutex though lock var is 0 and 'wait has ended'.

The threads are probably hung inside the operating system.

Have you had other unexplained crashes?

Any problems with the hardware?

Regards,

Heikki

There are no other unexplained crashes, I've kept a close eye on the hardware and it is performing as it should (and it has ECC memory too). Just to be sure I'm moving these database to another server and when I get the chance I'll run a memtest on the server.

Glenn,

have you now tested on another server?

Regards,

Heikki

No feedback was provided for this bug for over a month, so it is
being suspended automatically. If you are able to provide the
information that was originally requested, please do so and change
the status of the bug back to "Open".

Without seeing the stack traces of all executing threads during the hang or crash, it is impossible to tell if this is a duplicate of Bug #59733. But it could be.