Bug #44438 InnoDB: Assertion failure in thread 2467117968 in file rem0rec.c line 339
Submitted: 23 Apr 2009 15:55 Modified: 30 Jul 2009 15:45
Reporter: Dmitry Wonttell Email Updates:
Status: No Feedback Impact on me:
None 
Category:MySQL Server: InnoDB storage engine Severity:S1 (Critical)
Version:5.0.77-1 OS:Linux (CentOS)
Assigned to: CPU Architecture:Any
Tags: 339, assertion failure, crash, MySQL, rem0rec.c

[23 Apr 2009 15:55] Dmitry Wonttell
Description:
MySQL server crashes in a loop and can not recover. innodb_force_recovery=1 does not prevent server from crashing. As soon as we start up the server, it performs batch log entry apply to the db:

InnoDB: Starting an apply batch of log records to the database...
InnoDB: Progress in percents: 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99
InnoDB: Apply batch completed

and then crashes:

InnoDB: Assertion failure in thread 2467117968 in file rem0rec.c line 339
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: http://dev.mysql.com/doc/refman/5.0/en/forcing-recovery.html
InnoDB: about forcing recovery.
090423 18:35:07 - mysqld got signal 11 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help diagnose
the problem, but since we have already crashed, something is definitely wrong
and this may fail.

key_buffer_size=1048576
read_buffer_size=4194304
max_used_connections=0
max_connections=250
threads_connected=0
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_connections = 2049024 K
bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

thd=(nil)
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
Cannot determine thread, fp=0x930d1f08, backtrace may not be correct.
Stack range sanity check OK, backtrace follows:
0x818d3f8
0x8431317
0x842fa7c
0x8370959
0x8371a00
0x83e003b
0x83eac0d
0x83eaf8e
0x837067b
0x83707f4
0x834ee58
0x55a45b
0x4b1e5e
New value of fp=(nil) failed sanity check, terminating stack trace!
Please read http://dev.mysql.com/doc/mysql/en/using-stack-trace.html and follow instructions on how to resolve the stack trace. Resolved
stack trace is much more helpful in diagnosing the problem, so please do
resolve it
The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you find out what is causing the crash.

You do not even have to interact with server, it just keeps crashing and starting up again...

How to repeat:
Actually, no clue. I have never touched mysql system files, ibdata especially, this failure happened unexpectedly. We have innodb_file_per_table option set, and sometimes one of our tables grows over ~1 million rows and table's data file is being corrupted. The we had to use innodb_force_recovery=1, drop the table, and reimport the backup. The problem was away. But now things got worse, i can't even take over control of mysql server, it just keeps crashing.
[23 Apr 2009 16:57] Valeriy Kravchuk
Please, send the resolved stack trace and upload the entire error log (compressed). There was a similar bug report, http://bugs.mysql.com/bug.php?id=42768. I want to check if it is related.

Had you tried to use higher values for innodb_force_recovery option?
[23 Apr 2009 17:36] Dmitry Wonttell
Yes, i have tried all options from 0 to 3 for innodb_force_recovery, nothing helped, server kept crashing.

Here is a resolved stack trace:

0x81c0759 _Z10MYSQLparsePv + 73209
0x83ceb2c trx_sys_init_at_db_start + 828
0x83cd7f3 trx_rseg_get_on_id + 787
0x834dbbe srv_error_monitor_thread + 574
0x8325183 __db_join + 1603
0x833497e __ham_add_dup + 542
0x833596a __ham_get_meta + 106
0x832e178 __dbreg_setup + 248
0x830a01f __bam_copy + 447
0x836aea0 ibuf_init_at_db_start + 1584
0xb7fa1240 _end + -1349933116
0xb7ddc49e _end + -1351787998

I have made a backup of database by copying ibdata1, ib_logfile0, ib_logfile1 and database folder. I have made some tests by renaming files in database folder and i have found out, that the problem was caused by one of the table ibd or frm files (table named keyword_loc). This is the same table we had problems with before. This is a very high-volume table, this is the largest table in database having the largest index. If it will be necessary, i can send you these files (frm and ibd) for further investigation.
[24 Apr 2009 6:57] Valeriy Kravchuk
Had you tried to start with innodb_force_recovery=6? How large are your data and log files?
[24 Apr 2009 10:20] Dmitry Wonttell
No, i have not tried to use innodb_force_recovery=6.

The file sizes are:
563M    ./ibdata1
5.1M    ./ib_logfile0
5.1M    ./ib_logfile1

16K     ./keyword_loc.frm <-- this one causes the crashing
552K    ./keyword_loc.ibd <-- this one causes the crashing
[24 Apr 2009 10:48] Dmitry Wonttell
I have just tried innodb_force_recovery from 0 and up, and this time server stopped crashing at innodb_force_recovery=2. Yesterday server kept crashing having innodb_force_recovery=3.
[24 Apr 2009 10:52] Dmitry Wonttell
This is the crash report for today:

InnoDB: Assertion failure in thread 2466548624 in file rem0rec.c line 339
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: http://dev.mysql.com/doc/refman/5.0/en/forcing-recovery.html
InnoDB: about forcing recovery.
090424 13:46:52 - mysqld got signal 11 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help diagnose
the problem, but since we have already crashed, something is definitely wrong
and this may fail.

key_buffer_size=1048576
read_buffer_size=4194304
max_used_connections=0
max_connections=250
threads_connected=0
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_connections = 2049024 K
bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

thd=(nil)
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
Cannot determine thread, fp=0x93046f08, backtrace may not be correct.
Stack range sanity check OK, backtrace follows:
0x818d3f8
0x8431317
0x842fa7c
0x8370959
0x8371a00
0x83e003b
0x83eac0d
0x83eaf8e
0x837067b
0x83707f4
0x834ee58
0x55a45b
0x4b1e5e
New value of fp=(nil) failed sanity check, terminating stack trace!
Please read http://dev.mysql.com/doc/mysql/en/using-stack-trace.html and follow instructions on how to resolve the stack trace. Resolved
stack trace is much more helpful in diagnosing the problem, so please do
resolve it
The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you find out what is causing the crash.

Here is resolved stack trace:

0x818d3f8 handle_segfault + 792
0x8431317 rec_get_offsets_func + 375
0x842fa7c page_cur_search_with_match + 252
0x8370959 ibuf_contract + 329
0x8371a00 ibuf_merge_or_delete_for_page + 1680
0x83e003b buf_page_io_complete + 619
0x83eac0d buf_LRU_invalidate_tablespace + 1389
0x83eaf8e buf_read_ibuf_merge_pages + 270
0x837067b ibuf_delete_for_discarded_space + 5499
0x83707f4 ibuf_contract_for_n_pages + 52
0x834ee58 srv_master_thread + 2664
0x55a45b (?)
0x4b1e5e (?)
[29 Apr 2009 8:46] Sveta Smirnova
Thank you for the feedback.

Please try innodb_force_recovery=6 and check system log files to be sure this is not hardware failure.
[29 Apr 2009 11:35] Dmitry Wonttell
I tried innodb_force_recovery=6, server started up.

Used dmesg to look for any errors. There were no error reports regarding any hardware failure. Hard disks are functioning properly.
[30 Jun 2009 15:45] Miguel Solorzano
Is there still the issue and have you tried the last release?. Thanks in advance.
[30 Jul 2009 23:00] Bugs System
No feedback was provided for this bug for over a month, so it is
being suspended automatically. If you are able to provide the
information that was originally requested, please do so and change
the status of the bug back to "Open".