Bug #66751 mysqld got signal 11
Submitted: 10 Sep 2012 8:44 Modified: 15 Oct 2012 6:59
Reporter: Royee Tager Email Updates:
Status: No Feedback Impact on me:
None 
Category:MySQL Server Severity:S2 (Serious)
Version:5.1.63 OS:Linux (Debian 6.0.5)
Assigned to: CPU Architecture:Any

[10 Sep 2012 8:44] Royee Tager
Description:
mysql has crashed.

syslog contents (I will add the log file):
======================
Sep  9 07:32:02 mysql nrpe[309]: Handling the connection...
Sep  9 07:32:02 mysql nrpe[309]: Host is asking for command 'check_users' to be run...
Sep  9 07:32:02 mysql nrpe[309]: Running command: /usr/local/nagios/libexec/check_users -w 10 -c 20
Sep  9 07:32:02 mysql nrpe[309]: Command completed with return code 0 and output: USERS OK - 1 users currently logged in |users=1;10;20;0
Sep  9 07:32:02 mysql nrpe[309]: Return Code: 0, Output: USERS OK - 1 users currently logged in |users=1;10;20;0
Sep  9 07:32:59 mysql mysqld: 04:32:59 UTC - mysqld got signal 11 ;
Sep  9 07:32:59 mysql mysqld: This could be because you hit a bug. It is also possible that this binary
Sep  9 07:32:59 mysql mysqld: or one of the libraries it was linked against is corrupt, improperly built,
Sep  9 07:32:59 mysql mysqld: or misconfigured. This error can also be caused by malfunctioning hardware.
Sep  9 07:32:59 mysql mysqld: We will try our best to scrape up some info that will hopefully help
Sep  9 07:32:59 mysql mysqld: diagnose the problem, but since we have already crashed, 
Sep  9 07:32:59 mysql mysqld: something is definitely wrong and this may fail.
Sep  9 07:32:59 mysql mysqld: 
Sep  9 07:32:59 mysql mysqld: key_buffer_size=16777216
Sep  9 07:32:59 mysql mysqld: read_buffer_size=131072
Sep  9 07:32:59 mysql mysqld: max_used_connections=110
Sep  9 07:32:59 mysql mysqld: max_threads=151
Sep  9 07:32:59 mysql mysqld: thread_count=68
Sep  9 07:32:59 mysql mysqld: connection_count=68
Sep  9 07:32:59 mysql mysqld: It is possible that mysqld could use up to 
Sep  9 07:32:59 mysql mysqld: key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 346521 K  bytes of memory
Sep  9 07:32:59 mysql mysqld: Hope that's ok; if not, decrease some variables in the equation.
Sep  9 07:32:59 mysql mysqld: 
Sep  9 07:32:59 mysql mysqld: Thread pointer: 0x7f96882e9470
Sep  9 07:32:59 mysql mysqld: Attempting backtrace. You can use the following information to find out
Sep  9 07:32:59 mysql mysqld: where mysqld died. If you see no messages after this, something went
Sep  9 07:32:59 mysql mysqld: terribly wrong...
Sep  9 07:32:59 mysql mysqld: stack_bottom = 7f9697a20e88 thread_stack 0x30000
Sep  9 07:32:59 mysql mysqld: /usr/sbin/mysqld(my_print_stacktrace+0x29) [0x7f98d173e689]
Sep  9 07:32:59 mysql mysqld: /usr/sbin/mysqld(handle_fatal_signal+0x483) [0x7f98d1552a53]
Sep  9 07:32:59 mysql mysqld: /lib/libpthread.so.0(+0xeff0) [0x7f98d0ca4ff0]
Sep  9 07:32:59 mysql mysqld: /usr/sbin/mysqld(rec_get_offsets_func+0x169) [0x7f98d1680579]
Sep  9 07:32:59 mysql mysqld: /usr/sbin/mysqld(page_copy_rec_list_end_to_created_page+0x275) [0x7f98d166f6d5]
Sep  9 07:32:59 mysql mysqld: /usr/sbin/mysqld(page_copy_rec_list_end+0x15d) [0x7f98d1676e0d]
Sep  9 07:32:59 mysql mysqld: /usr/sbin/mysqld(page_move_rec_list_end+0x86) [0x7f98d1677b36]
Sep  9 07:32:59 mysql mysqld: /usr/sbin/mysqld(btr_page_split_and_insert+0x9a8) [0x7f98d16c6128]
Sep  9 07:32:59 mysql mysqld: /usr/sbin/mysqld(btr_cur_pessimistic_insert+0x197) [0x7f98d16091d7]
Sep  9 07:32:59 mysql mysqld: /usr/sbin/mysqld(+0x57c5ee) [0x7f98d164e5ee]
Sep  9 07:32:59 mysql mysqld: /usr/sbin/mysqld(ibuf_insert+0x121) [0x7f98d164ea71]
Sep  9 07:32:59 mysql mysqld: /usr/sbin/mysqld(btr_cur_search_to_nth_level+0x3b9) [0x7f98d160d739]
Sep  9 07:32:59 mysql mysqld: /usr/sbin/mysqld(row_ins_index_entry_low+0x13b) [0x7f98d168411b]
Sep  9 07:32:59 mysql mysqld: /usr/sbin/mysqld(row_ins+0x18f) [0x7f98d168559f]
Sep  9 07:32:59 mysql mysqld: /usr/sbin/mysqld(row_ins_step+0x129) [0x7f98d1685909]
Sep  9 07:32:59 mysql mysqld: /usr/sbin/mysqld(row_insert_for_mysql+0x388) [0x7f98d1687378]
Sep  9 07:32:59 mysql mysqld: /usr/sbin/mysqld(ha_innobase::write_row(unsigned char*)+0xe3) [0x7f98d16039d3]
Sep  9 07:32:59 mysql mysqld: /usr/sbin/mysqld(handler::ha_write_row(unsigned char*)+0x68) [0x7f98d1544dc8]
Sep  9 07:32:59 mysql mysqld: /usr/sbin/mysqld(write_record(THD*, st_table*, st_copy_info*)+0x6c) [0x7f98d14caaac]
Sep  9 07:32:59 mysql mysqld: /usr/sbin/mysqld(mysql_insert(THD*, TABLE_LIST*, List<Item>&, List<List<Item> >&, List<Item>&, List<Item>&, enum_duplicates, bool)+0xede) [0x7f98d14ceabe]
Sep  9 07:32:59 mysql mysqld: /usr/sbin/mysqld(mysql_execute_command(THD*)+0xbd9) [0x7f98d14572d9]
Sep  9 07:32:59 mysql mysqld: /usr/sbin/mysqld(mysql_parse(THD*, char*, unsigned int, char const**)+0x3fb) [0x7f98d145c1fb]
Sep  9 07:32:59 mysql mysqld: /usr/sbin/mysqld(dispatch_command(enum_server_command, THD*, char*, unsigned int)+0x115a) [0x7f98d145d36a]
Sep  9 07:32:59 mysql mysqld: /usr/sbin/mysqld(do_command(THD*)+0xea) [0x7f98d145dcaa]
Sep  9 07:32:59 mysql mysqld: /usr/sbin/mysqld(handle_one_connection+0x236) [0x7f98d144f796]
Sep  9 07:32:59 mysql mysqld: /lib/libpthread.so.0(+0x68ca) [0x7f98d0c9c8ca]
Sep  9 07:32:59 mysql mysqld: /lib/libc.so.6(clone+0x6d) [0x7f98cf7e492d]
Sep  9 07:32:59 mysql mysqld: 
Sep  9 07:32:59 mysql mysqld: Trying to get some variables.
Sep  9 07:32:59 mysql mysqld: Some pointers may be invalid and cause the dump to abort.
Sep  9 07:32:59 mysql mysqld: Query (7f968c0ca8b0): is an invalid pointer
Sep  9 07:32:59 mysql mysqld: Connection ID (thread ID): 5081837
Sep  9 07:32:59 mysql mysqld: Status: NOT_KILLED

How to repeat:
don't know
[10 Sep 2012 8:58] Royee Tager
syslog

Attachment: mysql_syslog.txt (text/plain), 23.79 KiB.

[13 Sep 2012 20:50] Royee Tager
Anyone?
[14 Sep 2012 19:58] Sveta Smirnova
Thank you for the report.

Do you meet this error on startup or during normal operations? Can you start the server? If not, have you tried option innodb-force-recovery?
[15 Sep 2012 6:54] Royee Tager
Hi Sveta,

Thanks for your answer.
Currently, the DB is up and running (after restoring the DB)

We would like to know why did the crash occur. Could you please investigate and advise regarding this issue?

Thanks in advance,

Royee Tager
[15 Sep 2012 6:59] Sveta Smirnova
Thank you for the feedback.

Crash happened when InnoDB tried to access page which number is out of bounds. This can happen either because corruption (in this case you need to check your disk and the fact nobody except MySQL server accesses its files) or because bug in MySQL code. But for latest case we need repeatable test case from you: actions, necessary to perform to repeat the crash.

I assume you have not such test case and corrupted data now. So I set status of this report to "Need feedback". Reopen it if crash happens again.
[16 Oct 2012 1:00] Bugs System
No feedback was provided for this bug for over a month, so it is
being suspended automatically. If you are able to provide the
information that was originally requested, please do so and change
the status of the bug back to "Open".