MySQL Bugs: #19539: Crash on InnoDB-only MySQL server

Bug #19539	Crash on InnoDB-only MySQL server
Submitted:	4 May 2006 15:57	Modified:	12 Feb 2007 8:58
Reporter:	Cuong Do	Email Updates:
Status:	No Feedback	Impact on me:	None
Category:	MySQL Server: InnoDB storage engine	Severity:	S2 (Serious)
Version:	4.1.18, 4.0.26	OS:	Linux (Suse Enterprise 9)
Assigned to:	Assigned Account	CPU Architecture:	Any

Description:
Under heavy load, a MySQL instance on a production database server crashed and restarted. The database server was doing greater than 10,000 queries per second.

Here's the information from the MySQL error log:

060503 20:41:59InnoDB: Assertion failure in thread 1237657952 in file ../include/sync0rw.ic line 364
InnoDB: Failing assertion: lock->reader_count > 0
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: http://dev.mysql.com/doc/mysql/en/Forcing_recovery.html
InnoDB: about forcing recovery.
InnoDB: Thread 1266960736 stopped in file row0sel.c line 2008
InnoDB: Thread 1165203808 stopped in file ../include/sync0sync.ic line 111
InnoDB: Thread 1263950176 stopped in file row0sel.c line 2008
mysqld got signal 11;

This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help diagnose
the problem, but since we have already crashed, something is definitely wrong
and this may fail.

key_buffer_size=33554432
read_buffer_size=1044480
max_used_connections=775
max_connections=1000
threads_connected=643
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_connections = 3100760 K
bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

InnoDB: Thread 1201129824 stopped in file row0sel.c line 2008
InnoDB: Thread 1136101728 stopped in file sync0arr.c line 336

Number of processes running now: 0

How to repeat:
This is a production server with heavy load. It handles about 90% short SELECT queries, and 10% small UPDATEs/INSERTs/DELETEs. All tables are InnoDB. It's difficult for me to describe a reproduction scenario, since this is the first crash of this kind I've seen. The system seems to have ample physical memory, and there are no process size limits.

The production server is running Suse Enterprise 9, with an official community AMD64 build (the system is 64-bit).

Thank you for a problem report. Please, send your my.cnf content. Any ideas on how to repeat are also welcomed.

You may try to use newer version, 4.1.19, but I am not sure that it will help.

Had it on 4.0.26, FC4, replication slave server too (at very high load).

I've seen this twice in the past two weeks.  MySQL 4.1.11 (based on the Debian package), Debian Sarge, AMD64.

Similarly, this happens to us under very heave read and write load.  All tables are innodb.

A thought - I never saw this crash until I moved to a 2x dual-core AMD setup (from 2x single-core) a month ago.  Since then I've seen it twice.

What hardware are other people who've seen this crash using?  I might guess that there's a race condition inside innodb which is only exposed when there are lots of CPUs and/or tightly coupled cores.

It now appears that was probably due to a libc bug in my distribution:

http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=314408

It has since been patched.

All reporters:

Please, try to repeat with a newer version, 4.1.22, and inform about the results. What glibc version do you use? Is this bug reperatable only on multi-processor/core AMD64 suystems?

No feedback was provided for this bug for over a month, so it is
being suspended automatically. If you are able to provide the
information that was originally requested, please do so and change
the status of the bug back to "Open".