Description:
So for the first time our production system produced the below assertion failure. The system came up without issues after mysqld automatically restarted itself and rolled back some transactions..
But why did this happen suddenly on a stable system?
Uptime is over 5 months, Mysqld was up for about 4 straight months with no other errors. Nothing in terms of file system errors, memory errors.
The system is a Ubuntu 10.4 64bit with ~16gb ram. .. running mysql 5.1.41 all Innodb tables.
can someone explain a little more what this assertion failure means more specifically?
=== error.log ===
120127 7:57:00 InnoDB: Assertion failure in thread 139886060934912 in file ../../../storage/innobase/rem/rem0rec.c line 337
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: http://dev.mysql.com/doc/refman/5.1/en/forcing-recovery.html
InnoDB: about forcing recovery.
120127 7:57:00 - mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help diagnose
the problem, but since we have already crashed, something is definitely wrong
and this may fail.
key_buffer_size=536870912
read_buffer_size=131072
max_used_connections=107
max_threads=151
threads_connected=49
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 854399 K
bytes of memory
Hope that's ok; if not, decrease some variables in the equation.
thd: 0x7f39ac188290
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0x7f39c2f88e58 thread_stack 0x30000
/usr/sbin/mysqld(my_print_stacktrace+0x29) [0x7f3cb33671a9]
/usr/sbin/mysqld(handle_segfault+0x404) [0x7f3cb3074884]
/lib/libpthread.so.0(+0xf8f0) [0x7f3cb28e58f0]
/lib/libc.so.6(gsignal+0x35) [0x7f3cb135fa75]
/lib/libc.so.6(abort+0x180) [0x7f3cb13635c0]
/usr/sbin/mysqld(rec_get_offsets_func+0xe0) [0x7f3cb32a9e50]
/usr/sbin/mysqld(row_search_for_mysql+0x56e) [0x7f3cb32be4ce]
/usr/sbin/mysqld(ha_innobase::general_fetch(unsigned char*, unsigned int, unsigned int)+0x7c) [0x7f3cb32271ec]
/usr/sbin/mysqld(+0x3ccb0e) [0x7f3cb30e1b0e]
/usr/sbin/mysqld(sub_select(JOIN*, st_join_table*, bool)+0x81) [0x7f3cb30dfeb1]
/usr/sbin/mysqld(+0x3cb3fd) [0x7f3cb30e03fd]
/usr/sbin/mysqld(JOIN::exec()+0xbac) [0x7f3cb30f55ec]
/usr/sbin/mysqld(mysql_select(THD*, Item***, TABLE_LIST*, unsigned int, List<Item>&, Item*, unsigned int, st_order*, st_order*, Item*, st_order*, unsigned long long, select_result*, st_select_lex_unit*, st_select_lex*)+0x142) [0x7f3cb30f1302]
/usr/sbin/mysqld(handle_select(THD*, st_lex*, select_result*, unsigned long)+0x174) [0x7f3cb30f6c04]
/usr/sbin/mysqld(+0x36d9fa) [0x7f3cb30829fa]
/usr/sbin/mysqld(mysql_execute_command(THD*)+0x526) [0x7f3cb3086806]
/usr/sbin/mysqld(mysql_parse(THD*, char const*, unsigned int, char const**)+0x35f) [0x7f3cb308bcbf]
/usr/sbin/mysqld(dispatch_command(enum_server_command, THD*, char*, unsigned int)+0xd69) [0x7f3cb308ca39]
/usr/sbin/mysqld(do_command(THD*)+0xea) [0x7f3cb308d61a]
/usr/sbin/mysqld(handle_one_connection+0x23d) [0x7f3cb307f64d]
/lib/libpthread.so.0(+0x69ca) [0x7f3cb28dc9ca]
/lib/libc.so.6(clone+0x6d) [0x7f3cb141270d]
Trying to get some variables.
Some pointers may be invalid and cause the dump to abort...
thd->query at 0x7f3ccb4021b0 = /* load collection com.lus111.db.bean.BnSource.inputSources */ select inputsourc0_.CHILD_SOURCE_UID as CHILD1_1_, inputsourc0_.PARENT_SOURCE_UID as PARENT2_1_, bnsource1_.UID as UID22_0_, bnsource1_.VERSION as VERSION22_0_, bnsource1_.MODIFIED_DATE as MODIFIED3_22_0_, bnsource1_.MODIFIED_BY as MODIFIED4_22_0_, bnsource1_.CREATED_DATE as CREATED5_22_0_, bnsource1_.CREATED_BY as CREATED6_22_0_, bnsource1_.IS_PUBLISHED as IS7_22_0_, bnsource1_.FORECAST_NAME_TRACKER as FORECAST8_22_0_, bnsource1_.CLIENT_UID as CLIENT9_22_0_, bnsource1_.COMPANY_UID as COMPANY10_22_0_, bnsource1_.LTUSER_UID as LTUSER11_22_0_, bnsource1_.FILE_UPLOAD_PDO_UID as FILE12_22_0_, bnsource1_.REPORT_UID as REPORT13_22_0_, bnsource1_.SOURCE_GENERATOR_UID as SOURCE14_22_0_ from Source_Inputs inputsourc0_ left outer join Source bnsource1_ on inputsourc0_.PARENT_SOURCE_UID=bnsource1_.UID where inputsourc0_.CHILD_SOURCE_UID=17950
thd->thread_id=264341
thd->killed=NOT_KILLED
The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you find out what is causing the crash.
pure virtual method called
terminate called without an active exception
120127 7:57:03 [Note] Plugin 'FEDERATED' is disabled.
InnoDB: Log scan progressed past the checkpoint lsn 224 4208337564
120127 7:57:06 InnoDB: Database was not shut down normally!
InnoDB: Starting crash recovery.
InnoDB: Reading tablespace information from the .ibd files...
InnoDB: Restoring possible half-written data pages from the doublewrite
InnoDB: buffer...
InnoDB: Doing recovery: scanned up to log sequence number 224 4213580288
=== end error.log ===
. then eventually it recovered with no further errors.
Do I need to worry more about this?
How to repeat:
Can't repeat it hasn't reoccurred as of yet.
