Bug #2636 MySql Crash
Submitted: 3 Feb 2004 19:00 Modified: 4 Feb 2004 18:15
Reporter: Xing Li Email Updates:
Status: Can't repeat Impact on me:
None 
Category:MySQL Server: InnoDB storage engine Severity:S2 (Serious)
Version:4.0.16 OS:Linux (RedHat 9)
Assigned to: Heikki Tuuri CPU Architecture:Any

[3 Feb 2004 19:00] Xing Li
Description:
Dual CPU setup with unmodified RedHat kernel. This is my mysql master server runing replication with one slave. Error happened on the master. Both are running 4.0.16 binary release.

Just suffered crash today and here is the resolve trace result:

0x8070670 handle_segfault + 420
0x8288b08 pthread_sighandler + 184
0x818158b btr_search_check_guess + 4123
0x818281a btr_search_guess_on_hash + 4562
0x8169f42 btr_cur_search_to_nth_level + 158
0x813cc5a row_sel_get_clust_rec_for_mysql + 114
0x8140ca2 row_search_for_mysql + 10142
0x80cb519 index_read__11ha_innobasePcPCcUi16ha_rkey_function + 673
0x80cb841 index_read_idx__11ha_innobasePcUiPCcUi16ha_rkey_function + 61
0x809d632 join_read_const__FP13st_join_table + 102
0x809d429 join_read_const_table__FP13st_join_tableP11st_position + 85
0x8095e3e make_join_statistics__FP4JOINP13st_table_listP4ItemP16st_dynamic_array + 2094
0x8094187 mysql_select__FP3THDP13st_table_listRt4List1Z4ItemP4ItemP8st_orderT4T3T4UlP13select_result + 2071
0x8093936 handle_select__FP3THDP6st_lexP13select_result + 102
0x807ae9a mysql_execute_command__Fv + 966
0x807ea75 mysql_parse__FP3THDPcUi + 153
0x8079fe3 dispatch_command__F19enum_server_commandP3THDPcUi + 1435
0x8079a3d do_command__FP3THD + 165
0x8079229 handle_one_connection + 641
0x82862bc pthread_start_thread + 220
0x82bba7a thread_start + 4

How to repeat:
Don't know what caused this.

Suggested fix:
None
[3 Feb 2004 19:05] Xing Li
Just some background info. Upgraded to 4.0.17 when it was released but suffered weird complete freeze of mysqld problem with zero error logging several times and after viewing the change log for inprogress 4.0.18, it looks like I might have hit the query "hang" bug mentioned in the changelog so waiting for 4.0.18 before upgrading version again.

Lastly, the above server's innodb file was "defragged", coverted to myisam + removed all innodb files + mysql auto regenerated new innodb files + converted tables back to innodb just 4 days ago under 4.0.16 so datafiles are relatively "fresh" and clean sort to speak.

The db is extremely active with innodb handling about 3 million records of about 500MB of data and myisam handing about 14+ million more at around 15.5GB.

After the crash, mysql apparently restarted itself.
[3 Feb 2004 19:12] Xing Li
Complete error log info related to this crash.

/usr/local/mysql/bin/mysqld: ready for connections.
Version: '4.0.16-standard-log'  socket: '/tmp/mysql.sock'  port: 22222
InnoDB: Error: trying to access a stray pointer 0
InnoDB: buf pool start is at 44c6c000, number of pages 65536
040202 18:50:02  InnoDB: Assertion failure in thread 73617422 in file ../../innobase/btr/../include/buf0buf.ic line 284
InnoDB: Failing assertion: 0
InnoDB: We intentionally generate a memory trap.
InnoDB: Send a detailed bug report to mysql@lists.mysql.com
mysqld got signal 11;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help diagnose
the problem, but since we have already crashed, something is definitely wrong
and this may fail.

key_buffer_size=268435456
read_buffer_size=2093056
max_used_connections=50
max_connections=50
threads_connected=3
It is possible that mysqld could use up to 
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_connections = 620343 K
bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

thd=0x8ae0b0b0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
Cannot determine thread, fp=0xbfdfe0e8, backtrace may not be correct.
InnoDB: Thread 73744403 stopped in file ha_innodb.cc line 396
Stack range sanity check OK, backtrace follows:
0x8070670
0x8288b08
0x818158b
0x818281a
0x8169f42
0x813cc5a
0x8140ca2
0x80cb519
0x80cb841
0x809d632
0x809d429
0x8095e3e
0x8094187
0x8093936
0x807ae9a
0x807ea75
0x8079fe3
0x8079a3d
0x8079229
0x82862bc
0x82bba7a
New value of fp=(nil) failed sanity check, terminating stack trace!
Please read http://www.mysql.com/doc/en/Using_stack_trace.html and follow instructions on how to resolve the stack trace. Resolved
stack trace is much more helpful in diagnosing the problem, so please do 
resolve it
Trying to get some variables.
Some pointers may be invalid and cause the dump to abort...
thd->query at 0x8b70f5a8 InnoDB: Thread 73728010 stopped in file ha_innodb.cc line 396
 is invalid pointer
thd->thread_id=3529459
The manual page at http://www.mysql.com/doc/en/Crashing.html contains
information that should help you find out what is causing the crash.
InnoDB: Thread 73723932 stopped in file ha_innodb.cc line 396

Number of processes running now: 0
040202 18:50:03  mysqld restarted
040202 18:50:06  InnoDB: Database was not shut down normally.
InnoDB: Starting recovery from log files...
InnoDB: Starting log scan based on checkpoint at
InnoDB: log sequence number 0 949346655
InnoDB: Doing recovery: scanned up to log sequence number 0 949347491
040202 18:50:06  InnoDB: Starting an apply batch of log records to the database...
InnoDB: Progress in percents: 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 
InnoDB: Apply batch completed
InnoDB: Last MySQL binlog file position 0 294203723, file name ./dev-bin.007
040202 18:50:07  InnoDB: Flushing modified pages from the buffer pool...
040202 18:50:07  InnoDB: Started
/usr/local/mysql/bin/mysqld: ready for connections.
Version: '4.0.16-standard-log'  socket: '/tmp/mysql.sock'  port: 22222
[4 Feb 2004 18:15] Michael Widenius
Not enough information was provided for us to be able
to handle this bug. Please re-read the instructions at
http://bugs.mysql.com/how-to-report.php

If you can provide more information, feel free to add it
to this bug and change the status back to 'Open'.

Thank you for your interest in MySQL.

Additional info:

Sorry, but the given information is not enough for us to be able to repeat the problem.

Please try 4.0.18 when it comes out.  If you still get a crash, please upload the used mysqld binary + the core file to:

ftp://support.mysql.com/pub/mysql/secret/

and we will try to find out what could cause this.

Another option is to try to follow the instructions in the MySQL manual and make a repeatable test case that we can use to find and fix the problem.