| Bug #44438 | InnoDB: Assertion failure in thread 2467117968 in file rem0rec.c line 339 | ||
|---|---|---|---|
| Submitted: | 23 Apr 2009 15:55 | Modified: | 30 Jul 2009 15:45 |
| Reporter: | Dmitry Wonttell | Email Updates: | |
| Status: | No Feedback | Impact on me: | |
| Category: | MySQL Server: InnoDB storage engine | Severity: | S1 (Critical) |
| Version: | 5.0.77-1 | OS: | Linux (CentOS) |
| Assigned to: | CPU Architecture: | Any | |
| Tags: | 339, assertion failure, crash, MySQL, rem0rec.c | ||
[23 Apr 2009 15:55]
Dmitry Wonttell
[23 Apr 2009 16:57]
Valeriy Kravchuk
Please, send the resolved stack trace and upload the entire error log (compressed). There was a similar bug report, http://bugs.mysql.com/bug.php?id=42768. I want to check if it is related. Had you tried to use higher values for innodb_force_recovery option?
[23 Apr 2009 17:36]
Dmitry Wonttell
Yes, i have tried all options from 0 to 3 for innodb_force_recovery, nothing helped, server kept crashing. Here is a resolved stack trace: 0x81c0759 _Z10MYSQLparsePv + 73209 0x83ceb2c trx_sys_init_at_db_start + 828 0x83cd7f3 trx_rseg_get_on_id + 787 0x834dbbe srv_error_monitor_thread + 574 0x8325183 __db_join + 1603 0x833497e __ham_add_dup + 542 0x833596a __ham_get_meta + 106 0x832e178 __dbreg_setup + 248 0x830a01f __bam_copy + 447 0x836aea0 ibuf_init_at_db_start + 1584 0xb7fa1240 _end + -1349933116 0xb7ddc49e _end + -1351787998 I have made a backup of database by copying ibdata1, ib_logfile0, ib_logfile1 and database folder. I have made some tests by renaming files in database folder and i have found out, that the problem was caused by one of the table ibd or frm files (table named keyword_loc). This is the same table we had problems with before. This is a very high-volume table, this is the largest table in database having the largest index. If it will be necessary, i can send you these files (frm and ibd) for further investigation.
[24 Apr 2009 6:57]
Valeriy Kravchuk
Had you tried to start with innodb_force_recovery=6? How large are your data and log files?
[24 Apr 2009 10:20]
Dmitry Wonttell
No, i have not tried to use innodb_force_recovery=6. The file sizes are: 563M ./ibdata1 5.1M ./ib_logfile0 5.1M ./ib_logfile1 16K ./keyword_loc.frm <-- this one causes the crashing 552K ./keyword_loc.ibd <-- this one causes the crashing
[24 Apr 2009 10:48]
Dmitry Wonttell
I have just tried innodb_force_recovery from 0 and up, and this time server stopped crashing at innodb_force_recovery=2. Yesterday server kept crashing having innodb_force_recovery=3.
[24 Apr 2009 10:52]
Dmitry Wonttell
This is the crash report for today: InnoDB: Assertion failure in thread 2466548624 in file rem0rec.c line 339 InnoDB: We intentionally generate a memory trap. InnoDB: Submit a detailed bug report to http://bugs.mysql.com. InnoDB: If you get repeated assertion failures or crashes, even InnoDB: immediately after the mysqld startup, there may be InnoDB: corruption in the InnoDB tablespace. Please refer to InnoDB: http://dev.mysql.com/doc/refman/5.0/en/forcing-recovery.html InnoDB: about forcing recovery. 090424 13:46:52 - mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. key_buffer_size=1048576 read_buffer_size=4194304 max_used_connections=0 max_connections=250 threads_connected=0 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_connections = 2049024 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. thd=(nil) Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... Cannot determine thread, fp=0x93046f08, backtrace may not be correct. Stack range sanity check OK, backtrace follows: 0x818d3f8 0x8431317 0x842fa7c 0x8370959 0x8371a00 0x83e003b 0x83eac0d 0x83eaf8e 0x837067b 0x83707f4 0x834ee58 0x55a45b 0x4b1e5e New value of fp=(nil) failed sanity check, terminating stack trace! Please read http://dev.mysql.com/doc/mysql/en/using-stack-trace.html and follow instructions on how to resolve the stack trace. Resolved stack trace is much more helpful in diagnosing the problem, so please do resolve it The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. Here is resolved stack trace: 0x818d3f8 handle_segfault + 792 0x8431317 rec_get_offsets_func + 375 0x842fa7c page_cur_search_with_match + 252 0x8370959 ibuf_contract + 329 0x8371a00 ibuf_merge_or_delete_for_page + 1680 0x83e003b buf_page_io_complete + 619 0x83eac0d buf_LRU_invalidate_tablespace + 1389 0x83eaf8e buf_read_ibuf_merge_pages + 270 0x837067b ibuf_delete_for_discarded_space + 5499 0x83707f4 ibuf_contract_for_n_pages + 52 0x834ee58 srv_master_thread + 2664 0x55a45b (?) 0x4b1e5e (?)
[29 Apr 2009 8:46]
Sveta Smirnova
Thank you for the feedback. Please try innodb_force_recovery=6 and check system log files to be sure this is not hardware failure.
[29 Apr 2009 11:35]
Dmitry Wonttell
I tried innodb_force_recovery=6, server started up. Used dmesg to look for any errors. There were no error reports regarding any hardware failure. Hard disks are functioning properly.
[30 Jun 2009 15:45]
MySQL Verification Team
Is there still the issue and have you tried the last release?. Thanks in advance.
[30 Jul 2009 23:00]
Bugs System
No feedback was provided for this bug for over a month, so it is being suspended automatically. If you are able to provide the information that was originally requested, please do so and change the status of the bug back to "Open".
