Bug #72237 MySQL Crash
Submitted: 4 Apr 2014 13:08 Modified: 7 May 2014 19:22
Reporter: Luis Joui Email Updates:
Status: No Feedback Impact on me:
None 
Category:MySQL Server Severity:S1 (Critical)
Version:5.5.35-0ubuntu0.12.04.2 OS:Linux (ubuntu0.12.04.2)
Assigned to: CPU Architecture:Any
Tags: btr0cur.c, crash

[4 Apr 2014 13:08] Luis Joui
Description:

We have Mysql Installed on a Virtual Server (VMWare) running over ubuntu Server 12.04.2. It works fine around 2 or 3 weeks and then crush always in btr0cur.c but in different lines. 

When we try to restart the sevice, then it go to stop and the error Log shows:

140404  9:37:13 [Warning] Using unique option prefix myisam-recover instead of myisam-recover-options is deprecated and will be removed in a future release. Please use the full name instead.
140404  9:37:13 [Note] Plugin 'FEDERATED' is disabled.
140404  9:37:13 InnoDB: The InnoDB memory heap is disabled
140404  9:37:13 InnoDB: Mutexes and rw_locks use GCC atomic builtins
140404  9:37:13 InnoDB: Compressed tables use zlib 1.2.3.4
140404  9:37:13 InnoDB: Initializing buffer pool, size = 256.0M
140404  9:37:13 InnoDB: Completed initialization of buffer pool
140404  9:37:13 InnoDB: highest supported file format is Barracuda.
InnoDB: The log sequence number in ibdata files does not match
InnoDB: the log sequence number in the ib_logfiles!
140404  9:37:13  InnoDB: Database was not shut down normally!
InnoDB: Starting crash recovery.
InnoDB: Reading tablespace information from the .ibd files...
InnoDB: Restoring possible half-written data pages from the doublewrite
InnoDB: buffer...
InnoDB: Last MySQL binlog file position 0 107, file name /var/log/mysql/mysql-bin.001686
140404  9:37:14  InnoDB: Waiting for the background threads to start
140404  9:37:15 InnoDB: 5.5.35 started; log sequence number 15782429751
140404  9:37:15 [Note] Recovering after a crash using /var/log/mysql/mysql-bin
140404  9:37:15 [Note] Starting crash recovery...
140404  9:37:15 [Note] Crash recovery finished.
140404  9:37:15 [Note] Server hostname (bind-address): '127.0.0.1'; port: 3306
140404  9:37:15 [Note]   - '127.0.0.1' resolves to '127.0.0.1';
140404  9:37:15 [Note] Server socket created on IP: '127.0.0.1'.
140404  9:37:16 [Warning] Neither --relay-log nor --relay-log-index were used; so replication may break when this MySQL server acts as a slave and has his hostname changed!! Please use '--relay-log=mysqld-relay-bin' to avoid this problem.
140404  9:37:16 [Note] Slave SQL thread initialized, starting replication in log 'mysql-bin.000077' at position 80032659, relay log './mysqld-relay-bin.000062' position: 79106336
140404  9:37:16 [Note] Slave I/O thread: connected to master 'wisereplicador@127.0.0.1:33066',replication started in log 'mysql-bin.000077' at position 80037761
140404  9:37:16 [Note] Event Scheduler: Loaded 0 events
140404  9:37:16 [Note] /usr/sbin/mysqld: ready for connections.
Version: '5.5.35-0ubuntu0.12.04.2-log'  socket: '/var/run/mysqld/mysqld.sock'  port: 3306  (Ubuntu)
140404  9:37:18  InnoDB: Assertion failure in thread 139757630732032 in file btr0cur.c line 277
InnoDB: Failing assertion: btr_page_get_next(get_block->frame, mtr) == page_get_page_no(page)
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: http://dev.mysql.com/doc/refman/5.5/en/forcing-innodb-recovery.html
InnoDB: about forcing recovery.
12:37:18 UTC - mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed, 
something is definitely wrong and this may fail.

key_buffer_size=134217728
read_buffer_size=524288
max_used_connections=9
max_threads=150
thread_count=9
connection_count=9
It is possible that mysqld could use up to 
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 516800 K  bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0x7f1b98000990
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 7f1bdbef2820 thread_stack 0x31000
/usr/sbin/mysqld(my_print_stacktrace+0x29)[0x7f1bdc5d4719]
/usr/sbin/mysqld(handle_fatal_signal+0x483)[0x7f1bdc499ae3]
/lib/x86_64-linux-gnu/libpthread.so.0(+0xfcb0)[0x7f1bdb1e3cb0]
/lib/x86_64-linux-gnu/libc.so.6(gsignal+0x35)[0x7f1bda84e425]
/lib/x86_64-linux-gnu/libc.so.6(abort+0x17b)[0x7f1bda851b8b]
/usr/sbin/mysqld(+0x630a9d)[0x7f1bdc6aba9d]
/usr/sbin/mysqld(+0x6314eb)[0x7f1bdc6ac4eb]
/usr/sbin/mysqld(+0x6cdbc8)[0x7f1bdc748bc8]
/usr/sbin/mysqld(+0x6d1255)[0x7f1bdc74c255]
/usr/sbin/mysqld(+0x6d1690)[0x7f1bdc74c690]
/usr/sbin/mysqld(+0x5e8016)[0x7f1bdc663016]
/usr/sbin/mysqld(+0x5d5c1a)[0x7f1bdc650c1a]
/usr/sbin/mysqld(_ZN7handler12ha_write_rowEPh+0x68)[0x7f1bdc4a1738]
/usr/sbin/mysqld(_ZN14Rows_log_event9write_rowEPK14Relay_log_infob+0x119)[0x7f1bdc54a949]
/usr/sbin/mysqld(_ZN20Write_rows_log_event11do_exec_rowEPK14Relay_log_info+0x20)[0x7f1bdc54ac60]
/usr/sbin/mysqld(_ZN14Rows_log_event14do_apply_eventEPK14Relay_log_info+0x20a)[0x7f1bdc540daa]
/usr/sbin/mysqld(_Z26apply_event_and_update_posP9Log_eventP3THDP14Relay_log_info+0x135)[0x7f1bdc31b825]
/usr/sbin/mysqld(handle_slave_sql+0xe35)[0x7f1bdc31cde5]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x7e9a)[0x7f1bdb1dbe9a]
/lib/x86_64-linux-gnu/libc.so.6(clone+0x6d)[0x7f1bda90c3fd]

Trying to get some variables.
Some pointers may be invalid and cause the dump to abort.
Query (0): is an invalid pointer
Connection ID (thread ID): 1
Status: NOT_KILLED

The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you find out what is causing the crash.

How to repeat:
We don't know
[4 Apr 2014 19:46] Sveta Smirnova
Thank you for the report.

Please check your disk on errors: assertion complains about pages, which are read from the disk.
[7 Apr 2014 15:06] Luis Joui
The server is a VPS and our service provider said "the disk has no physical problems (physical disk of their virtual machines is a RAID storage, which they have monitored)"
Is there any way to define/detect the disk problem to tell it to our service provider?
Is there a MySQL configuration to avoid this problem? or at least avoid the database corruption?
[7 Apr 2014 19:22] Sveta Smirnova
Thank you the feedback.

Not, you need to use OS tools to check if disk is corrupted or not.

Regarding to the question about how to avoid corruption: yes, there are InnoDB options, which can help, particularly innodb_doublewrite and innodb_flush_log_at_trx_commit (http://dev.mysql.com/doc/refman/5.5/en/mysqld-option-tables.html)

Since you cannot check if disk was corrupted or not please send us full error log file and try to identify if repeating scenario leads to crash, so we can be sure this is not MySQL bug.
[8 May 2014 1:00] Bugs System
No feedback was provided for this bug for over a month, so it is
being suspended automatically. If you are able to provide the
information that was originally requested, please do so and change
the status of the bug back to "Open".