InnoDB: Page directory corruption: infimum not pointed to 120102 8:34:32 InnoDB: Page dump in ascii and hex (16384 bytes): len 16384; hex (...) 120102 8:34:32 InnoDB: Page checksum 2582938661, prior-to-4.0.14-form checksum 1446043165 InnoDB: stored checksum 2582938661, prior-to-4.0.14-form stored checksum 1446043165 InnoDB: Page lsn 51 1002274109, low 4 bytes of lsn at page end 1002274109 InnoDB: Page number (if stored to page already) 44271, InnoDB: space id (if created with >= MySQL-4.1.1 and stored already) 0 InnoDB: Page directory corruption: supremum not pointed to 120102 8:34:32 InnoDB: Page dump in ascii and hex (16384 bytes): (...) 120102 8:34:32 InnoDB: Page checksum 2582938661, prior-to-4.0.14-form checksum 1446043165 InnoDB: stored checksum 2582938661, prior-to-4.0.14-form stored checksum 1446043165 InnoDB: Page lsn 51 1002274109, low 4 bytes of lsn at page end 1002274109 InnoDB: Page number (if stored to page already) 44271, InnoDB: space id (if created with >= MySQL-4.1.1 and stored already) 0 120102 8:34:32 InnoDB: Assertion failure in thread 2920119152 in file ../../../storage/innobase/rem/rem0rec.c line 337 InnoDB: We intentionally generate a memory trap. InnoDB: Submit a detailed bug report to http://bugs.mysql.com. InnoDB: If you get repeated assertion failures or crashes, even InnoDB: immediately after the mysqld startup, there may be InnoDB: corruption in the InnoDB tablespace. Please refer to InnoDB: http://dev.mysql.com/doc/refman/5.1/en/forcing-recovery.html InnoDB: about forcing recovery. 120102 8:34:32 - mysqld got signal 6 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. key_buffer_size=16777216 read_buffer_size=131072 max_used_connections=107 max_threads=160 threads_connected=15 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 365560 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. thd: 0xb1904478 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0xae0d736c thread_stack 0x20000 /usr/sbin/mysqld(my_print_stacktrace+0x2d) [0xb76c5cbd] /usr/sbin/mysqld(handle_segfault+0x494) [0xb7391854] [0xb710c400] /lib/tls/i686/cmov/libc.so.6(abort+0x182) [0xb6e16a82] /usr/sbin/mysqld(rec_get_offsets_func+0xbb) [0xb75f4adb] /usr/sbin/mysqld(page_cur_search_with_match+0x2e6) [0xb75e36b6] /usr/sbin/mysqld(btr_cur_search_to_nth_level+0x5ae) [0xb7579c8e] /usr/sbin/mysqld(row_ins_index_entry_low+0x123) [0xb75f8ca3] /usr/sbin/mysqld(row_ins+0x19e) [0xb75fa47e] /usr/sbin/mysqld(row_ins_step+0x110) [0xb75fa780] /usr/sbin/mysqld(row_insert_for_mysql+0x37e) [0xb760254e] /usr/sbin/mysqld(ha_innobase::write_row(unsigned char*)+0xf9) [0xb756e309] /usr/sbin/mysqld(handler::ha_write_row(unsigned char*)+0x6d) [0xb749fd3d] /usr/sbin/mysqld(write_record(THD*, st_table*, st_copy_info*)+0x7b) [0xb741eaeb] /usr/sbin/mysqld(mysql_insert(THD*, TABLE_LIST*, List&, List >&, List&, List&, enum_duplicates, bool)+0x1122) [0xb74237c2] /usr/sbin/mysqld(mysql_execute_command(THD*)+0xc85) [0xb73a4c95] /usr/sbin/mysqld(mysql_parse(THD*, char const*, unsigned int, char const**)+0x3ae) [0xb73ac45e] /usr/sbin/mysqld(dispatch_command(enum_server_command, THD*, char*, unsigned int)+0xd96) [0xb73ad206] /usr/sbin/mysqld(do_command(THD*)+0xf3) [0xb73ae563] /usr/sbin/mysqld(handle_one_connection+0x2a0) [0xb739d050] /lib/tls/i686/cmov/libpthread.so.0(+0x596e) [0xb70f096e] /lib/tls/i686/cmov/libc.so.6(clone+0x5e) [0xb6eb6a4e] Trying to get some variables. Some pointers may be invalid and cause the dump to abort... thd->query at 0xb826c3c8 = INSERT INTO sys_log (userid,type,action,error,details_nr,details,log_data,tablename,recuid,IP,tstamp,event_pid,NEWid,workspace) VALUES ('36','1','2','0','10','Record \'%s\' (%s) was updated. (Online).','a:2:{i:0;s:0:\"\";i:1;s:15:\"tt_content:2314\";}','tt_content','2314','172.30.128.136','1325489671','1238','','0') thd->thread_id=54374 thd->killed=NOT_KILLED The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. 120102 8:34:32 [Note] Plugin 'FEDERATED' is disabled. InnoDB: Log scan progressed past the checkpoint lsn 51 3164629765 120102 8:34:32 InnoDB: Database was not shut down normally! InnoDB: Starting crash recovery. InnoDB: Reading tablespace information from the .ibd files... InnoDB: Restoring possible half-written data pages from the doublewrite InnoDB: buffer... InnoDB: Doing recovery: scanned up to log sequence number 51 3164630613 120102 8:34:33 InnoDB: Starting an apply batch of log records to the database... InnoDB: Progress in percents: 88 89 90 91 92 93 94 95 96 97 98 99 InnoDB: Apply batch completed InnoDB: Last MySQL binlog file position 0 167607, file name /var/log/mysql/mysql-bin.001226 120102 8:34:33 InnoDB: Started; log sequence number 51 3164630613 120102 8:34:33 [Note] Event Scheduler: Loaded 0 events 120102 8:34:33 [Note] /usr/sbin/mysqld: ready for connections. Version: '5.1.41-3ubuntu12.10' socket: '/var/run/mysqld/mysqld.sock' port: 3306 (Ubuntu) InnoDB: Next record offset is nonsensical 64000 in record at offset 101 InnoDB: rec address 0xb3bf0065, first buffer frame 0xb39d8000 InnoDB: buffer pool high end 0xb41d8000, buf fix count 2 120102 10:51:05 InnoDB: Page dump in ascii and hex (16384 bytes): 120102 10:51:06 InnoDB: Page checksum 2582938661, prior-to-4.0.14-form checksum 1446043165 InnoDB: stored checksum 2582938661, prior-to-4.0.14-form stored checksum 1446043165 InnoDB: Page lsn 51 1002274109, low 4 bytes of lsn at page end 1002274109 InnoDB: Page number (if stored to page already) 44271, InnoDB: space id (if created with >= MySQL-4.1.1 and stored already) 0 120102 10:51:06 InnoDB: Assertion failure in thread 2931039088 in file ../../../storage/innobase/include/page0page.ic line 591 InnoDB: We intentionally generate a memory trap. InnoDB: Submit a detailed bug report to http://bugs.mysql.com. InnoDB: If you get repeated assertion failures or crashes, even InnoDB: immediately after the mysqld startup, there may be InnoDB: corruption in the InnoDB tablespace. Please refer to InnoDB: http://dev.mysql.com/doc/refman/5.1/en/forcing-recovery.html InnoDB: about forcing recovery. 120102 10:51:06 - mysqld got signal 6 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. key_buffer_size=16777216 read_buffer_size=131072 max_used_connections=57 max_threads=160 threads_connected=17 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 365560 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. thd: 0xb8512148 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0xaeb4136c thread_stack 0x20000 /usr/sbin/mysqld(my_print_stacktrace+0x2d) [0xb7695cbd] /usr/sbin/mysqld(handle_segfault+0x494) [0xb7361854] [0xb70dc400] /lib/tls/i686/cmov/libc.so.6(abort+0x182) [0xb6de6a82] /usr/sbin/mysqld(page_cur_open_on_rnd_user_rec+0x2ac) [0xb75b5ebc] /usr/sbin/mysqld(btr_estimate_number_of_different_key_vals+0x4aa) [0xb7546cea] /usr/sbin/mysqld(dict_update_statistics_low+0x76) [0xb7560f06] /usr/sbin/mysqld(dict_update_statistics+0x19) [0xb7560fb9] /usr/sbin/mysqld(ha_innobase::info(unsigned int)+0x5bc) [0xb753cc3c] /usr/sbin/mysqld(+0x3abbcd) [0xb74a5bcd] /usr/sbin/mysqld(get_all_tables(THD*, TABLE_LIST*, Item*)+0x351) [0xb74ab571] /usr/sbin/mysqld(get_schema_tables_result(JOIN*, enum_schema_table_state)+0x1eb) [0xb74999eb] /usr/sbin/mysqld(JOIN::exec()+0x5cb) [0xb73e96ab] /usr/sbin/mysqld(mysql_select(THD*, Item***, TABLE_LIST*, unsigned int, List&, Item*, unsigned int, st_order*, st_order*, Item*, st_order*, unsigned long long, select_result*, st_select_lex_unit*, st_select_lex*)+0xe7) [0xb73eb5f7] /usr/sbin/mysqld(handle_select(THD*, st_lex*, select_result*, unsigned long)+0x164) [0xb73ec0a4] /usr/sbin/mysqld(+0x27667f) [0xb737067f] /usr/sbin/mysqld(mysql_execute_command(THD*)+0x24fc) [0xb737650c] /usr/sbin/mysqld(mysql_parse(THD*, char const*, unsigned int, char const**)+0x3ae) [0xb737c45e] /usr/sbin/mysqld(dispatch_command(enum_server_command, THD*, char*, unsigned int)+0xd96) [0xb737d206] /usr/sbin/mysqld(do_command(THD*)+0xf3) [0xb737e563] /usr/sbin/mysqld(handle_one_connection+0x2a0) [0xb736d050] /lib/tls/i686/cmov/libpthread.so.0(+0x596e) [0xb70c096e] /lib/tls/i686/cmov/libc.so.6(clone+0x5e) [0xb6e86a4e] Trying to get some variables. Some pointers may be invalid and cause the dump to abort... thd->query at 0xb8264978 = SHOW KEYS FROM `sys_log` thd->thread_id=3348 thd->killed=NOT_KILLED The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. 120102 10:51:06 [Note] Plugin 'FEDERATED' is disabled. InnoDB: Log scan progressed past the checkpoint lsn 51 3180882163 120102 10:51:06 InnoDB: Database was not shut down normally! InnoDB: Starting crash recovery. InnoDB: Reading tablespace information from the .ibd files... InnoDB: Restoring possible half-written data pages from the doublewrite InnoDB: buffer... InnoDB: Doing recovery: scanned up to log sequence number 51 3180882190 120102 10:51:06 InnoDB: Starting an apply batch of log records to the database... InnoDB: Progress in percents: 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 InnoDB: Apply batch completed InnoDB: Last MySQL binlog file position 0 167607, file name /var/log/mysql/mysql-bin.001226 120102 10:51:06 InnoDB: Started; log sequence number 51 3180882190 120102 10:51:06 [Note] Event Scheduler: Loaded 0 events 120102 10:51:06 [Note] /usr/sbin/mysqld: ready for connections. Version: '5.1.41-3ubuntu12.10' socket: '/var/run/mysqld/mysqld.sock' port: 3306 (Ubuntu) InnoDB: Next record offset is nonsensical 64000 in record at offset 101 InnoDB: rec address 0xb3a08065, first buffer frame 0xb39a8000 InnoDB: buffer pool high end 0xb41a8000, buf fix count 2 120102 10:51:49 InnoDB: Page dump in ascii and hex (16384 bytes): (...) 120102 10:51:49 InnoDB: Page checksum 2582938661, prior-to-4.0.14-form checksum 1446043165 InnoDB: stored checksum 2582938661, prior-to-4.0.14-form stored checksum 1446043165 InnoDB: Page lsn 51 1002274109, low 4 bytes of lsn at page end 1002274109 InnoDB: Page number (if stored to page already) 44271, InnoDB: space id (if created with >= MySQL-4.1.1 and stored already) 0 120102 10:51:49 InnoDB: Assertion failure in thread 2980252528 in file ../../../storage/innobase/include/page0page.ic line 591 InnoDB: We intentionally generate a memory trap. InnoDB: Submit a detailed bug report to http://bugs.mysql.com. InnoDB: If you get repeated assertion failures or crashes, even InnoDB: immediately after the mysqld startup, there may be InnoDB: corruption in the InnoDB tablespace. Please refer to InnoDB: http://dev.mysql.com/doc/refman/5.1/en/forcing-recovery.html InnoDB: about forcing recovery. 120102 10:51:49 - mysqld got signal 6 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. key_buffer_size=16777216 read_buffer_size=131072 max_used_connections=16 max_threads=160 threads_connected=15 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 365560 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. thd: 0xb9613b70 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0xb1a3036c thread_stack 0x20000 /usr/sbin/mysqld(my_print_stacktrace+0x2d) [0xb7667cbd] /usr/sbin/mysqld(handle_segfault+0x494) [0xb7333854] [0xb70ae400] /lib/tls/i686/cmov/libc.so.6(abort+0x182) [0xb6db8a82] /usr/sbin/mysqld(page_cur_open_on_rnd_user_rec+0x2ac) [0xb7587ebc] /usr/sbin/mysqld(btr_estimate_number_of_different_key_vals+0x4aa) [0xb7518cea] /usr/sbin/mysqld(dict_update_statistics_low+0x76) [0xb7532f06] /usr/sbin/mysqld(dict_update_statistics+0x19) [0xb7532fb9] /usr/sbin/mysqld(ha_innobase::info(unsigned int)+0x5bc) [0xb750ec3c] /usr/sbin/mysqld(+0x3ae1a8) [0xb747a1a8] /usr/sbin/mysqld(get_all_tables(THD*, TABLE_LIST*, Item*)+0xc62) [0xb747de82] /usr/sbin/mysqld(get_schema_tables_result(JOIN*, enum_schema_table_state)+0x1eb) [0xb746b9eb] /usr/sbin/mysqld(JOIN::exec()+0x5cb) [0xb73bb6ab] /usr/sbin/mysqld(mysql_select(THD*, Item***, TABLE_LIST*, unsigned int, List&, Item*, unsigned int, st_order*, st_order*, Item*, st_order*, unsigned long long, select_result*, st_select_lex_unit*, st_select_lex*)+0xe7) [0xb73bd5f7] /usr/sbin/mysqld(handle_select(THD*, st_lex*, select_result*, unsigned long)+0x164) [0xb73be0a4] /usr/sbin/mysqld(+0x27667f) [0xb734267f] /usr/sbin/mysqld(mysql_execute_command(THD*)+0x24fc) [0xb734850c] /usr/sbin/mysqld(mysql_parse(THD*, char const*, unsigned int, char const**)+0x3ae) [0xb734e45e] /usr/sbin/mysqld(dispatch_command(enum_server_command, THD*, char*, unsigned int)+0xd96) [0xb734f206] /usr/sbin/mysqld(do_command(THD*)+0xf3) [0xb7350563] /usr/sbin/mysqld(handle_one_connection+0x2a0) [0xb733f050] /lib/tls/i686/cmov/libpthread.so.0(+0x596e) [0xb709296e] /lib/tls/i686/cmov/libc.so.6(clone+0x5e) [0xb6e58a4e] Trying to get some variables. Some pointers may be invalid and cause the dump to abort... thd->query at 0xb94f2ea0 = SHOW TABLE STATUS FROM `t3_swb_2008` thd->thread_id=13 thd->killed=NOT_KILLED The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. 120102 10:51:49 [Note] Plugin 'FEDERATED' is disabled. InnoDB: The log sequence number in ibdata files does not match InnoDB: the log sequence number in the ib_logfiles! 120102 10:51:49 InnoDB: Database was not shut down normally! InnoDB: Starting crash recovery. InnoDB: Reading tablespace information from the .ibd files... InnoDB: Restoring possible half-written data pages from the doublewrite InnoDB: buffer... InnoDB: Last MySQL binlog file position 0 167607, file name /var/log/mysql/mysql-bin.001226 120102 10:51:49 InnoDB: Started; log sequence number 51 3180883447 120102 10:51:49 [Note] Event Scheduler: Loaded 0 events 120102 10:51:49 [Note] /usr/sbin/mysqld: ready for connections. Version: '5.1.41-3ubuntu12.10' socket: '/var/run/mysqld/mysqld.sock' port: 3306 (Ubuntu)