Bug #64172 Mysql crashed once with an “Assertion failure" rem0rec.c line 337
Submitted: 30 Jan 2012 19:24 Modified: 29 Feb 2012 19:01
Reporter: Tom G Email Updates:
Status: No Feedback Impact on me:
Category:MySQL Server: InnoDB storage engine Severity:S3 (Non-critical)
Version: 5.1.41 OS:Linux (Ubuntu 10.04)
Assigned to: CPU Architecture:Any
Tags: crash, innodb, MySQL

[30 Jan 2012 19:24] Tom G
So for the first time our production system produced the below assertion failure. The system came up without issues after mysqld automatically restarted itself and rolled back some transactions.. 

But why did this happen suddenly on a stable system? 

Uptime is over 5 months, Mysqld was up for about 4 straight months with no other errors. Nothing in terms of file system errors, memory errors. 

The system is a Ubuntu 10.4 64bit with ~16gb ram. .. running mysql 5.1.41 all Innodb tables. 

can someone explain a little more what this assertion failure means more specifically? 

=== error.log === 

120127 7:57:00 InnoDB: Assertion failure in thread 139886060934912 in file ../../../storage/innobase/rem/rem0rec.c line 337 
InnoDB: We intentionally generate a memory trap. 
InnoDB: Submit a detailed bug report to http://bugs.mysql.com. 
InnoDB: If you get repeated assertion failures or crashes, even 
InnoDB: immediately after the mysqld startup, there may be 
InnoDB: corruption in the InnoDB tablespace. Please refer to 
InnoDB: http://dev.mysql.com/doc/refman/5.1/en/forcing-recovery.html 
InnoDB: about forcing recovery. 
120127 7:57:00 - mysqld got signal 6 ; 
This could be because you hit a bug. It is also possible that this binary 
or one of the libraries it was linked against is corrupt, improperly built, 
or misconfigured. This error can also be caused by malfunctioning hardware. 
We will try our best to scrape up some info that will hopefully help diagnose 
the problem, but since we have already crashed, something is definitely wrong 
and this may fail. 

It is possible that mysqld could use up to 
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 854399 K 
bytes of memory 
Hope that's ok; if not, decrease some variables in the equation. 

thd: 0x7f39ac188290 
Attempting backtrace. You can use the following information to find out 
where mysqld died. If you see no messages after this, something went 
terribly wrong... 
stack_bottom = 0x7f39c2f88e58 thread_stack 0x30000 
/usr/sbin/mysqld(my_print_stacktrace+0x29) [0x7f3cb33671a9] 
/usr/sbin/mysqld(handle_segfault+0x404) [0x7f3cb3074884] 
/lib/libpthread.so.0(+0xf8f0) [0x7f3cb28e58f0] 
/lib/libc.so.6(gsignal+0x35) [0x7f3cb135fa75] 
/lib/libc.so.6(abort+0x180) [0x7f3cb13635c0] 
/usr/sbin/mysqld(rec_get_offsets_func+0xe0) [0x7f3cb32a9e50] 
/usr/sbin/mysqld(row_search_for_mysql+0x56e) [0x7f3cb32be4ce] 
/usr/sbin/mysqld(ha_innobase::general_fetch(unsigned char*, unsigned int, unsigned int)+0x7c) [0x7f3cb32271ec] 
/usr/sbin/mysqld(+0x3ccb0e) [0x7f3cb30e1b0e] 
/usr/sbin/mysqld(sub_select(JOIN*, st_join_table*, bool)+0x81) [0x7f3cb30dfeb1] 
/usr/sbin/mysqld(+0x3cb3fd) [0x7f3cb30e03fd] 
/usr/sbin/mysqld(JOIN::exec()+0xbac) [0x7f3cb30f55ec] 
/usr/sbin/mysqld(mysql_select(THD*, Item***, TABLE_LIST*, unsigned int, List<Item>&, Item*, unsigned int, st_order*, st_order*, Item*, st_order*, unsigned long long, select_result*, st_select_lex_unit*, st_select_lex*)+0x142) [0x7f3cb30f1302] 
/usr/sbin/mysqld(handle_select(THD*, st_lex*, select_result*, unsigned long)+0x174) [0x7f3cb30f6c04] 
/usr/sbin/mysqld(+0x36d9fa) [0x7f3cb30829fa] 
/usr/sbin/mysqld(mysql_execute_command(THD*)+0x526) [0x7f3cb3086806] 
/usr/sbin/mysqld(mysql_parse(THD*, char const*, unsigned int, char const**)+0x35f) [0x7f3cb308bcbf] 
/usr/sbin/mysqld(dispatch_command(enum_server_command, THD*, char*, unsigned int)+0xd69) [0x7f3cb308ca39] 
/usr/sbin/mysqld(do_command(THD*)+0xea) [0x7f3cb308d61a] 
/usr/sbin/mysqld(handle_one_connection+0x23d) [0x7f3cb307f64d] 
/lib/libpthread.so.0(+0x69ca) [0x7f3cb28dc9ca] 
/lib/libc.so.6(clone+0x6d) [0x7f3cb141270d] 
Trying to get some variables. 
Some pointers may be invalid and cause the dump to abort... 
thd->query at 0x7f3ccb4021b0 = /* load collection com.lus111.db.bean.BnSource.inputSources */ select inputsourc0_.CHILD_SOURCE_UID as CHILD1_1_, inputsourc0_.PARENT_SOURCE_UID as PARENT2_1_, bnsource1_.UID as UID22_0_, bnsource1_.VERSION as VERSION22_0_, bnsource1_.MODIFIED_DATE as MODIFIED3_22_0_, bnsource1_.MODIFIED_BY as MODIFIED4_22_0_, bnsource1_.CREATED_DATE as CREATED5_22_0_, bnsource1_.CREATED_BY as CREATED6_22_0_, bnsource1_.IS_PUBLISHED as IS7_22_0_, bnsource1_.FORECAST_NAME_TRACKER as FORECAST8_22_0_, bnsource1_.CLIENT_UID as CLIENT9_22_0_, bnsource1_.COMPANY_UID as COMPANY10_22_0_, bnsource1_.LTUSER_UID as LTUSER11_22_0_, bnsource1_.FILE_UPLOAD_PDO_UID as FILE12_22_0_, bnsource1_.REPORT_UID as REPORT13_22_0_, bnsource1_.SOURCE_GENERATOR_UID as SOURCE14_22_0_ from Source_Inputs inputsourc0_ left outer join Source bnsource1_ on inputsourc0_.PARENT_SOURCE_UID=bnsource1_.UID where inputsourc0_.CHILD_SOURCE_UID=17950 
The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains 
information that should help you find out what is causing the crash. 
pure virtual method called 
terminate called without an active exception 
120127 7:57:03 [Note] Plugin 'FEDERATED' is disabled. 
InnoDB: Log scan progressed past the checkpoint lsn 224 4208337564 
120127 7:57:06 InnoDB: Database was not shut down normally! 
InnoDB: Starting crash recovery. 
InnoDB: Reading tablespace information from the .ibd files... 
InnoDB: Restoring possible half-written data pages from the doublewrite 
InnoDB: buffer... 
InnoDB: Doing recovery: scanned up to log sequence number 224 4213580288 
=== end error.log ===  

. then eventually it recovered with no further errors.

Do I need to worry more about this?

How to repeat:
Can't repeat it hasn't reoccurred as of yet.
[31 Jan 2012 8:27] Valeriy Kravchuk
There are some bug reports that look somewhat related, like bug #55935, bug #44438 etc. Please, make sure there are no suspicious messages in /var/log/messages etc, as sometimes assertions like this are related to hardware failures.
[31 Jan 2012 15:56] Tom G
nothing indicative of any faults , file system errors in any of the logs.

So far it has been 4 days and it has been stable so far.

Is there a way of finding out what caused this from the above log?
[31 Jan 2012 19:01] Sveta Smirnova
Thank you for the feedback.

Will it crash if you run affected query ( select inputsourc0_.CHILD_SOURCE_UID as
CHILD1_1_, inputsourc0_.PARENT_SOURCE_UID as PARENT2_1_, bnsource1_.UID as UID22_0_,
bnsource1_.VERSION as VERSION22_0_, bnsource1_.MODIFIED_DATE as MODIFIED3_22_0_,
...) isolated? E.g. open new connection and run it.
[1 Mar 2012 1:00] Bugs System
No feedback was provided for this bug for over a month, so it is
being suspended automatically. If you are able to provide the
information that was originally requested, please do so and change
the status of the bug back to "Open".