InnoDB: ###### Diagnostic info printed to the standard error stream InnoDB: Error in pages 1220160 and 258049 of index "PRIMARY" of table "x"."x" InnoDB: broken FIL_PAGE_NEXT or FIL_PAGE_PREV links 120426 8:07:10 InnoDB: Page dump in ascii and hex (16384 bytes): len 16384; hex 6d9a5... InnoDB: End of page dump 120426 8:07:10 InnoDB: Page checksum 1838829962 (32bit_calc: 2985562315), prior-to-4.0.14-form checksum 4108579494 InnoDB: stored checksum 1838829962, prior-to-4.0.14-form stored checksum 4108579494 InnoDB: Page lsn 344 1470213567, low 4 bytes of lsn at page end 1470213567 InnoDB: Page number (if stored to page already) 1220160, InnoDB: space id (if created with >= MySQL-4.1.1 and stored already) 0 InnoDB: Page may be an index page where index id is 14074 InnoDB: (index "PRIMARY" of table "x"."x") 120426 8:07:10 InnoDB: Page dump in ascii and hex (16384 bytes): len 16384; hex c642e5... InnoDB: End of page dump 120426 8:07:10 InnoDB: Page checksum 3326272848 (32bit_calc: 2178245854), prior-to-4.0.14-form checksum 568955370 InnoDB: stored checksum 3326272848, prior-to-4.0.14-form stored checksum 568955370 InnoDB: Page lsn 337 1831943029, low 4 bytes of lsn at page end 1831943029 InnoDB: Page number (if stored to page already) 258049, InnoDB: space id (if created with >= MySQL-4.1.1 and stored already) 0 InnoDB: Page may be an index page where index id is 14074 InnoDB: (index "PRIMARY" of table "x"."x") 120427 0:40:46 InnoDB: Assertion failure in thread 1251584320 in file btr0cur.c line 314 InnoDB: Failing assertion: btr_page_get_prev(get_block->frame, mtr) == page_get_page_no(page) InnoDB: We intentionally generate a memory trap. InnoDB: Submit a detailed bug report to http://bugs.mysql.com. InnoDB: If you get repeated assertion failures or crashes, even InnoDB: immediately after the mysqld startup, there may be InnoDB: corruption in the InnoDB tablespace. Please refer to InnoDB: http://dev.mysql.com/doc/refman/5.5/en/forcing-innodb-recovery.html InnoDB: about forcing recovery. 22:40:46 UTC - mysqld got signal 6 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. key_buffer_size=629145600 read_buffer_size=2097152 max_used_connections=3 max_threads=900 thread_count=0 connection_count=0 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 9841417 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x20d3a320 ttempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 4a99a0f8 thread_stack 0x40000 /usr/sbin/mysqld(my_print_stacktrace+0x35)[0x7a42e5] /usr/sbin/mysqld(handle_fatal_signal+0x4a4)[0x67fca4] /lib64/libpthread.so.0[0x3f1ba0ebe0] /lib64/libc.so.6(gsignal+0x35)[0x3f1ae30285] /lib64/libc.so.6(abort+0x110)[0x3f1ae31d30] /usr/sbin/mysqld[0x8257fe] /usr/sbin/mysqld[0x8296f0] /usr/sbin/mysqld[0x8d6b94] /usr/sbin/mysqld[0x8d90f9] /usr/sbin/mysqld[0x7cf876] /usr/sbin/mysqld[0x7bc801] /usr/sbin/mysqld(_ZN7handler12ha_write_rowEPh+0x5f)[0x6853cf] /usr/sbin/mysqld(_ZN14Rows_log_event9write_rowEPK14Relay_log_infob+0xf2)[0x721562] /usr/sbin/mysqld(_ZN20Write_rows_log_event11do_exec_rowEPK14Relay_log_info+0x22)[0x721842] /usr/sbin/mysqld(_ZN14Rows_log_event14do_apply_eventEPK14Relay_log_info+0x217)[0x723197] /usr/sbin/mysqld(_Z26apply_event_and_update_posP9Log_eventP3THDP14Relay_log_info+0x125)[0x527c25] /usr/sbin/mysqld[0x52c6e9] /usr/sbin/mysqld(handle_slave_sql+0x919)[0x52da79] /lib64/libpthread.so.0[0x3f1ba0677d] /lib64/libc.so.6(clone+0x6d)[0x3f1aed325d] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0): is an invalid pointer Connection ID (thread ID): 2 Status: NOT_KILLED The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. 120427 00:40:47 mysqld_safe Number of processes running now: 0 120427 00:40:47 mysqld_safe mysqld restarted 120427 0:40:47 [Note] Flashcache bypass: disabled 120427 0:40:47 [Note] Flashcache setup error is : ioctl failed 120427 0:40:47 [Note] Plugin 'FEDERATED' is disabled. 120427 0:40:47 InnoDB: The InnoDB memory heap is disabled 120427 0:40:47 InnoDB: Mutexes and rw_locks use GCC atomic builtins 120427 0:40:47 InnoDB: Compressed tables use zlib 1.2.3 120427 0:40:47 InnoDB: Using Linux native AIO 120427 0:40:47 InnoDB: Initializing buffer pool, size = 6.8G 120427 0:40:48 InnoDB: Completed initialization of buffer pool 120427 0:40:48 InnoDB: highest supported file format is Barracuda. InnoDB: Log scan progressed past the checkpoint lsn 1482817469366 120427 0:40:48 InnoDB: Database was not shut down normally! InnoDB: Starting crash recovery. InnoDB: Reading tablespace information from the .ibd files... InnoDB: Restoring possible half-written data pages from the doublewrite InnoDB: buffer... InnoDB: Doing recovery: scanned up to log sequence number 1482822711808 InnoDB: Doing recovery: scanned up to log sequence number 1482827954688 InnoDB: Doing recovery: scanned up to log sequence number 1482833197568 InnoDB: Doing recovery: scanned up to log sequence number 1482838440448 InnoDB: Doing recovery: scanned up to log sequence number 1482843683328 InnoDB: Doing recovery: scanned up to log sequence number 1482848926208 InnoDB: Doing recovery: scanned up to log sequence number 1482854169088 InnoDB: Doing recovery: scanned up to log sequence number 1482859411968 InnoDB: Doing recovery: scanned up to log sequence number 1482864654848 InnoDB: Doing recovery: scanned up to log sequence number 1482869897728 InnoDB: Doing recovery: scanned up to log sequence number 1482875140608 InnoDB: Doing recovery: scanned up to log sequence number 1482879985444 120427 0:40:53 InnoDB: Starting an apply batch of log records to the database... InnoDB: Progress in percents: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 InnoDB: Apply batch completed InnoDB: In a MySQL replication slave the last master binlog file InnoDB: position 0 15740822, file name mysql-bin.001743 InnoDB: and relay log file InnoDB: position 0 15740968, file name /var/lib/mysql-bin/hnsl3-relay-bin.004688 InnoDB: Last MySQL binlog file position 0 118893870, file name /var/lib/mysql-bin/mysql-bin.000042 120427 0:40:58 InnoDB: Restoring buffer pool pages from ib_lru_dump 120427 0:40:58 InnoDB: Waiting for the background threads to start 120427 0:40:59 Percona XtraDB (http://www.percona.com) 1.1.8-rel25.1 started; log sequence number 1482879985444 120427 0:40:59 [Note] Recovering after a crash using /var/lib/mysql-bin/mysql-bin 120427 0:40:59 [Note] Starting crash recovery... 120427 0:40:59 [Note] Crash recovery finished. 120427 0:41:00 [Note] Slave SQL thread initialized, starting replication in log 'mysql-bin.001743' at position 15742388, relay log '/var/lib/mysql-bin/hnsl3-relay-bin.004688' position: 15742534 120427 0:41:00 [Note] Slave I/O thread: connected to master 'x@192.168.7.8:3306',replication started in log 'mysql-bin.001743' at position 15785887 120427 0:41:00 [Note] Event Scheduler: Loaded 0 events 120427 0:41:00 [Note] /usr/sbin/mysqld: ready for connections. Version: '5.5.21-55-log' socket: '/var/lib/mysql/mysql.sock' port: 3306 Percona Server (GPL), Release rel25.1, Revision 234 120427 0:41:01 InnoDB: Assertion failure in thread 1252129088 in file btr0cur.c line 314 InnoDB: Failing assertion: btr_page_get_prev(get_block->frame, mtr) == page_get_page_no(page) InnoDB: We intentionally generate a memory trap. InnoDB: Submit a detailed bug report to http://bugs.mysql.com. InnoDB: If you get repeated assertion failures or crashes, even InnoDB: immediately after the mysqld startup, there may be InnoDB: corruption in the InnoDB tablespace. Please refer to InnoDB: http://dev.mysql.com/doc/refman/5.5/en/forcing-innodb-recovery.html InnoDB: about forcing recovery. 22:41:01 UTC - mysqld got signal 6 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. and again and again ... with innodb_force_recovery 1 to 4 never ending 120427 16:13:52 InnoDB: Waiting for the background threads to start my innodb conf innodb_buffer_pool_size = 7000M innodb_flush_log_at_trx_commit = 2 innodb_log_file_size = 64M innodb_additional_mem_pool_size = 16M innodb_old_blocks_time=1000 after slave recovery from master show table status like 'x'; +----------+--------+---------+------------+----------+----------------+-------------+-----------------+--------------+-----------+----------------+---------------------+-------------+------------+-----------------+----------+----------------+---------+ | Name | Engine | Version | Row_format | Rows | Avg_row_length | Data_length | Max_data_length | Index_length | Data_free | Auto_increment | Create_time | Update_time | Check_time | Collation | Checksum | Create_options | Comment | +----------+--------+---------+------------+----------+----------------+-------------+-----------------+--------------+-----------+----------------+---------------------+-------------+------------+-----------------+----------+----------------+---------+ | xxxxxxxx | InnoDB | 10 | Compact | 45395275 | 46 | 2101346304 | 0 | 1146093568 | 17825792 | NULL | 2012-05-02 13:34:23 | NULL | NULL | utf8_general_ci | NULL | | | +----------+--------+---------+------------+----------+----------------+-------------+-----------------+--------------+-----------+----------------+---------------------+-------------+------------+-----------------+----------+----------------+---------+