Bug #50419 Server Crashes with : mysqld got signal 6
Submitted: 18 Jan 2010 15:47 Modified: 18 Jan 2010 20:13
Reporter: Eric Monchanin Email Updates:
Status: Duplicate Impact on me:
None 
Category:MySQL Server Severity:S1 (Critical)
Version:5.1.39-ndb-7.0.9-cluster-gpl OS:Linux (CentOS 5.3)
Assigned to: CPU Architecture:Any

[18 Jan 2010 15:47] Eric Monchanin
Description:
I am trying to run an "old" database set, and I noticed using some tables crashed mysqld.

My database holds 33 tables, and doing anything on 2 of them triggers a crash with the following output.

100118 10:02:11  InnoDB: Error: (1500) Couldn't read the MAX(user_id) autoinc value from the index (PRIMARY).
100118 10:02:11  InnoDB: Assertion failure in thread 2965437328 in file handler/ha_innodb.cc line 2738
InnoDB: Failing assertion: error == DB_SUCCESS
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: http://dev.mysql.com/doc/refman/5.1/en/forcing-recovery.html
InnoDB: about forcing recovery.
100118 10:02:11 - mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help diagnose
the problem, but since we have already crashed, something is definitely wrong
and this may fail.

key_buffer_size=67108864
read_buffer_size=1048576
max_used_connections=1
max_threads=151
threads_connected=1
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 839619 K
bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

thd: 0xa449108
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0xb0c0f3a8 thread_stack 0x30000
/usr/local/mysql-cluster-7.0.9/libexec/mysqld(my_print_stacktrace+0x2d) [0x85c8d5d]
/usr/local/mysql-cluster-7.0.9/libexec/mysqld(handle_segfault+0x37f) [0x824bdcf]
[0x203420]
/lib/libc.so.6(abort+0x101) [0x388701]
/usr/local/mysql-cluster-7.0.9/libexec/mysqld [0x842baa1]
/usr/local/mysql-cluster-7.0.9/libexec/mysqld(handler::ha_open(st_table*, char const*, int, int)+0x38) [0x83452f8]
/usr/local/mysql-cluster-7.0.9/libexec/mysqld(open_table_from_share(THD*, st_table_share*, char const*, unsigned int, unsigned int, unsigned int, st_table*, open_table_mode)+0x5b8) [0x82a9928]
/usr/local/mysql-cluster-7.0.9/libexec/mysqld [0x82a1905]
/usr/local/mysql-cluster-7.0.9/libexec/mysqld(open_table(THD*, TABLE_LIST*, st_mem_root*, bool*, unsigned int)+0x58a) [0x82a4e5a]
/usr/local/mysql-cluster-7.0.9/libexec/mysqld(open_tables(THD*, TABLE_LIST**, unsigned int*, unsigned int)+0x60f) [0x82a5c6f]
/usr/local/mysql-cluster-7.0.9/libexec/mysqld(open_and_lock_tables_derived(THD*, TABLE_LIST*, bool)+0x38) [0x82a5fb8]
/usr/local/mysql-cluster-7.0.9/libexec/mysqld [0x8257ec6]
/usr/local/mysql-cluster-7.0.9/libexec/mysqld(mysql_execute_command(THD*)+0x33ed) [0x825e64d]
/usr/local/mysql-cluster-7.0.9/libexec/mysqld(mysql_parse(THD*, char const*, unsigned int, char const**)+0x20f) [0x82652ef]
/usr/local/mysql-cluster-7.0.9/libexec/mysqld(dispatch_command(enum_server_command, THD*, char*, unsigned int)+0x44f) [0x826574f]
/usr/local/mysql-cluster-7.0.9/libexec/mysqld(do_command(THD*)+0xe1) [0x82669b1]
/usr/local/mysql-cluster-7.0.9/libexec/mysqld(handle_one_connection+0x64d) [0x8254e5d]
/lib/libpthread.so.0 [0x4b173b]
/lib/libc.so.6(clone+0x5e) [0x42fcfe]
Trying to get some variables.
Some pointers may be invalid and cause the dump to abort...
thd->query at 0xa4c14c8 = SELECT `admin_users`.*, (CASE WHEN `password` = 'toto' THEN 1 ELSE 0 END) AS `zend_auth_credential_match` FROM `admin_users` WHERE (`username` = 'toto')
thd->thread_id=9
thd->killed=NOT_KILLED
The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you find out what is causing the crash.
100118 10:02:11 mysqld_safe Number of processes running now: 0
100118 10:02:11 mysqld_safe mysqld restarted
100118 10:02:11 [Note] Plugin 'FEDERATED' is disabled.
100118 10:02:11 [Note] Plugin 'ndbcluster' is disabled.
InnoDB: The log sequence number in ibdata files does not match
InnoDB: the log sequence number in the ib_logfiles!
100118 10:02:11  InnoDB: Database was not shut down normally!
InnoDB: Starting crash recovery.
InnoDB: Reading tablespace information from the .ibd files...
InnoDB: Restoring possible half-written data pages from the doublewrite
InnoDB: buffer...
100118 10:02:11  InnoDB: Started; log sequence number 0 2320785
InnoDB: !!! innodb_force_recovery is set to 1 !!!
100118 10:02:11 [Note] Event Scheduler: Loaded 0 events
100118 10:02:11 [Note] /usr/local/mysql-cluster-7.0.9/libexec/mysqld: ready for connections.
Version: '5.1.39-ndb-7.0.9-cluster-gpl'  socket: '/var/lib/mysql/mysql.sock'  port: 3306  MySQL Cluster Server (GPL)

How to repeat:
I can repeat it every time I use those 2 tables :

SELECT, UPDATE, DELETE
CHECK TABLE
LOCK TABLES
[18 Jan 2010 15:50] Eric Monchanin
forgot a few details :
version mysql-cluster-gpl-7.0.9

compiled from source.

Same issue with mysql-cluster-gpl-7.0.5
[18 Jan 2010 17:14] Valeriy Kravchuk
Please, try to do a clean shutdown of MySQL server, then remove/comment out setting of innodb_force_recovery to 1 in your my.cnf, start server again and, if it starts properly, try to execute the same statements with the same InnoDB tables. Inform about the results.
[18 Jan 2010 18:25] Eric Monchanin
I started / stopped mysqld with the
innodb_force_recovery=1

Then I commented it out, and restarted the server.
Got the same problem, here is the dump :

100118 13:23:24  InnoDB: Error: (1500) Couldn't read the MAX(user_id) autoinc value from the index (PRIMARY).
100118 13:23:24  InnoDB: Assertion failure in thread 3024288656 in file handler/ha_innodb.cc line 2738
InnoDB: Failing assertion: error == DB_SUCCESS
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: http://dev.mysql.com/doc/refman/5.1/en/forcing-recovery.html
InnoDB: about forcing recovery.
100118 13:23:24 - mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help diagnose
the problem, but since we have already crashed, something is definitely wrong
and this may fail.

key_buffer_size=8384512
read_buffer_size=131072
max_used_connections=1
max_threads=151
threads_connected=1
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 337726 K
bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

thd: 0xab7fb08
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0xb442f3a8 thread_stack 0x30000
/usr/local/mysql/libexec/mysqld(my_print_stacktrace+0x2d) [0x85c8d5d]
/usr/local/mysql/libexec/mysqld(handle_segfault+0x37f) [0x824bdcf]
[0x27c420]
/lib/libc.so.6(abort+0x101) [0x388701]
/usr/local/mysql/libexec/mysqld [0x842baa1]
/usr/local/mysql/libexec/mysqld(handler::ha_open(st_table*, char const*, int, int)+0x38) [0x83452f8]
/usr/local/mysql/libexec/mysqld(open_table_from_share(THD*, st_table_share*, char const*, unsigned int, unsigned int, unsigned int, st_table*, open_table_mode)+0x5b8) [0x82a9928]
/usr/local/mysql/libexec/mysqld [0x82a1905]
/usr/local/mysql/libexec/mysqld(open_table(THD*, TABLE_LIST*, st_mem_root*, bool*, unsigned int)+0x58a) [0x82a4e5a]
/usr/local/mysql/libexec/mysqld(open_tables(THD*, TABLE_LIST**, unsigned int*, unsigned int)+0x60f) [0x82a5c6f]
/usr/local/mysql/libexec/mysqld(open_and_lock_tables_derived(THD*, TABLE_LIST*, bool)+0x38) [0x82a5fb8]
/usr/local/mysql/libexec/mysqld(mysql_execute_command(THD*)+0x5203) [0x8260463]
/usr/local/mysql/libexec/mysqld(mysql_parse(THD*, char const*, unsigned int, char const**)+0x20f) [0x82652ef]
/usr/local/mysql/libexec/mysqld(dispatch_command(enum_server_command, THD*, char*, unsigned int)+0x44f) [0x826574f]
/usr/local/mysql/libexec/mysqld(do_command(THD*)+0xe1) [0x82669b1]
/usr/local/mysql/libexec/mysqld(handle_one_connection+0x64d) [0x8254e5d]
/lib/libpthread.so.0 [0x4b173b]
/lib/libc.so.6(clone+0x5e) [0x42fcfe]
Trying to get some variables.
Some pointers may be invalid and cause the dump to abort...
thd->query at 0xabcbc98 = LOCK TABLES `admin_users` READ /*!32311 LOCAL */
thd->thread_id=1
thd->killed=NOT_KILLED
The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you find out what is causing the crash.
100118 13:23:24 mysqld_safe Number of processes running now: 0
100118 13:23:24 mysqld_safe mysqld restarted
100118 13:23:24 [Note] Plugin 'FEDERATED' is disabled.
100118 13:23:24 [Note] Plugin 'ndbcluster' is disabled.
100118 13:23:24  InnoDB: Started; log sequence number 0 2320795
100118 13:23:24 [Note] Event Scheduler: Loaded 0 events
100118 13:23:24 [Note] /usr/local/mysql/libexec/mysqld: ready for connections.
Version: '5.1.39-ndb-7.0.9-cluster-gpl'  socket: '/var/lib/mysql/mysql.sock'  port: 3306  MySQL Cluster Server (GPL)
[18 Jan 2010 20:13] Sveta Smirnova
Please do not submit the same bug more than once. An existing bug report already describes this very problem. Even if you feel that your issue is somewhat different, the resolution is likely
to be the same. Because of this, we hope you add your comments to the original bug instead.

Thank you for your interest in MySQL.

Duplicate of bug #44030 which is fixed in 5.1.41-ndb-7.0.11