060626 17:58:33 mysqld started InnoDB: Warning: we did not need to do crash recovery, but log scan InnoDB: progressed past the checkpoint lsn 576 61161632 up to lsn 576 61161662 060626 17:58:38 InnoDB: Started; log sequence number 576 61161632 /usr/local/mysql/bin/mysqld: ready for connections. Version: '4.1.18-standard-log' socket: '/tmp/mysql.sock' port: 3306 MySQL Community Edition - Standard (GPL) InnoDB: Error: thd->transaction.all.innodb_active_trans == 0 InnoDB: but trx->conc_state != TRX_NOT_STARTED InnoDB: Error: thd->transaction.all.innodb_active_trans == 0 InnoDB: but trx->conc_state != TRX_NOT_STARTED InnoDB: Error: thd->transaction.all.innodb_active_trans == 0 InnoDB: but trx->conc_state != TRX_NOT_STARTED InnoDB: Error: thd->transaction.all.innodb_active_trans == 0 InnoDB: but trx->conc_state != TRX_NOT_STARTED InnoDB: Error: thd->transaction.all.innodb_active_trans == 0 InnoDB: but trx->conc_state != TRX_NOT_STARTED InnoDB: Error: thd->transaction.all.innodb_active_trans == 0 InnoDB: but trx->conc_state != TRX_NOT_STARTED InnoDB: Error: thd->transaction.all.innodb_active_trans == 0 InnoDB: but trx->conc_state != TRX_NOT_STARTED InnoDB: Error: thd->transaction.all.innodb_active_trans == 0 InnoDB: but trx->conc_state != TRX_NOT_STARTED InnoDB: Error: thd->transaction.all.innodb_active_trans == 0 InnoDB: but trx->conc_state != TRX_NOT_STARTED InnoDB: Error: thd->transaction.all.innodb_active_trans == 0 InnoDB: but trx->conc_state != TRX_NOT_STARTED InnoDB: Error: thd->transaction.all.innodb_active_trans == 0 InnoDB: but trx->conc_state != TRX_NOT_STARTED InnoDB: Error: thd->transaction.all.innodb_active_trans == 0 InnoDB: but trx->conc_state != TRX_NOT_STARTED InnoDB: Error: thd->transaction.all.innodb_active_trans == 0 InnoDB: but trx->conc_state != TRX_NOT_STARTED InnoDB: Error: thd->transaction.all.innodb_active_trans == 0 InnoDB: but trx->conc_state != TRX_NOT_STARTED InnoDB: Error: thd->transaction.all.innodb_active_trans == 0 InnoDB: but trx->conc_state != TRX_NOT_STARTED InnoDB: Error: thd->transaction.all.innodb_active_trans == 0 InnoDB: but trx->conc_state != TRX_NOT_STARTED InnoDB: Error: thd->transaction.all.innodb_active_trans == 0 InnoDB: but trx->conc_state != TRX_NOT_STARTED InnoDB: Error: thd->transaction.all.innodb_active_trans == 0 InnoDB: but trx->conc_state != TRX_NOT_STARTED InnoDB: Error: thd->transaction.all.innodb_active_trans == 0 InnoDB: but trx->conc_state != TRX_NOT_STARTED InnoDB: Error: thd->transaction.all.innodb_active_trans == 0 InnoDB: but trx->conc_state != TRX_NOT_STARTED 060712 10:21:06 InnoDB: Error: a record lock wait happens in a dictionary operation! InnoDB: Table name `SYS_TABLES`. InnoDB: Submit a detailed bug report to http://bugs.mysql.com 060712 10:21:06InnoDB: Assertion failure in thread 52381696 in file trx0trx.c line 296 InnoDB: Failing assertion: UT_LIST_GET_LEN(trx->signals) == 0 InnoDB: We intentionally generate a memory trap. InnoDB: Submit a detailed bug report to http://bugs.mysql.com. InnoDB: If you get repeated assertion failures or crashes, even InnoDB: immediately after the mysqld startup, there may be InnoDB: corruption in the InnoDB tablespace. Please refer to InnoDB: http://dev.mysql.com/doc/mysql/en/Forcing_recovery.html InnoDB: about forcing recovery. InnoDB: Thread 54434816 stopped in file sync0arr.c line 336 InnoDB: Thread 52360192 stopped in file os0sync.c line 501 InnoDB: Thread 50459648 stopped in file sync0arr.c line 336 InnoDB: Thread 50455552 stopped in file os0sync.c line 501 mysqld got signal 10; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. key_buffer_size=5242880 read_buffer_size=1044480 max_used_connections=43 max_connections=100 threads_connected=20 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_connections = 926319 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. 060712 10:21:10 mysqld restarted 060712 10:21:13 InnoDB: Database was not shut down normally! InnoDB: Starting crash recovery. InnoDB: Reading tablespace information from the .ibd files... InnoDB: Restoring possible half-written data pages from the doublewrite InnoDB: buffer... 060712 10:21:13 InnoDB: Starting log scan based on checkpoint at InnoDB: log sequence number 666 956417828. InnoDB: Doing recovery: scanned up to log sequence number 666 961660416 InnoDB: Doing recovery: scanned up to log sequence number 666 966903296 InnoDB: Doing recovery: scanned up to log sequence number 666 972146176 InnoDB: Doing recovery: scanned up to log sequence number 666 977389056 InnoDB: Doing recovery: scanned up to log sequence number 666 982631936 InnoDB: Doing recovery: scanned up to log sequence number 666 987874816 InnoDB: Doing recovery: scanned up to log sequence number 666 993117696 InnoDB: Doing recovery: scanned up to log sequence number 666 998360576 InnoDB: Doing recovery: scanned up to log sequence number 666 1003603456 InnoDB: Doing recovery: scanned up to log sequence number 666 1008846336 InnoDB: Doing recovery: scanned up to log sequence number 666 1014089216 InnoDB: Doing recovery: scanned up to log sequence number 666 1019332096 InnoDB: Doing recovery: scanned up to log sequence number 666 1024574976 InnoDB: Doing recovery: scanned up to log sequence number 666 1029817856 InnoDB: Doing recovery: scanned up to log sequence number 666 1035060736 InnoDB: Doing recovery: scanned up to log sequence number 666 1040303616 InnoDB: Doing recovery: scanned up to log sequence number 666 1045546496 InnoDB: Doing recovery: scanned up to log sequence number 666 1050789376 InnoDB: Doing recovery: scanned up to log sequence number 666 1056032256 InnoDB: Doing recovery: scanned up to log sequence number 666 1061275136 InnoDB: Doing recovery: scanned up to log sequence number 666 1066518016 InnoDB: Doing recovery: scanned up to log sequence number 666 1071760896 InnoDB: Doing recovery: scanned up to log sequence number 666 1077003776 InnoDB: Doing recovery: scanned up to log sequence number 666 1082246656 InnoDB: Doing recovery: scanned up to log sequence number 666 1087489536 InnoDB: Doing recovery: scanned up to log sequence number 666 1092732416 InnoDB: Doing recovery: scanned up to log sequence number 666 1097975296 InnoDB: Doing recovery: scanned up to log sequence number 666 1103218176 InnoDB: Doing recovery: scanned up to log sequence number 666 1108461056 InnoDB: Doing recovery: scanned up to log sequence number 666 1113703936 InnoDB: Doing recovery: scanned up to log sequence number 666 1118946816 InnoDB: Doing recovery: scanned up to log sequence number 666 1124189696 InnoDB: Doing recovery: scanned up to log sequence number 666 1129432576 InnoDB: Doing recovery: scanned up to log sequence number 666 1134675456 InnoDB: Doing recovery: scanned up to log sequence number 666 1139918336 InnoDB: Doing recovery: scanned up to log sequence number 666 1145161216 InnoDB: Doing recovery: scanned up to log sequence number 666 1150404096 InnoDB: Doing recovery: scanned up to log sequence number 666 1155646976 InnoDB: Doing recovery: scanned up to log sequence number 666 1160889856 InnoDB: Doing recovery: scanned up to log sequence number 666 1166132736 InnoDB: Doing recovery: scanned up to log sequence number 666 1171375616 InnoDB: Doing recovery: scanned up to log sequence number 666 1176618496 InnoDB: Doing recovery: scanned up to log sequence number 666 1181861376 InnoDB: Doing recovery: scanned up to log sequence number 666 1187104256 InnoDB: Doing recovery: scanned up to log sequence number 666 1192347136 InnoDB: Doing recovery: scanned up to log sequence number 666 1197590016 InnoDB: Doing recovery: scanned up to log sequence number 666 1202832896 InnoDB: Doing recovery: scanned up to log sequence number 666 1208075776 InnoDB: Doing recovery: scanned up to log sequence number 666 1213318656 InnoDB: Doing recovery: scanned up to log sequence number 666 1218561536 InnoDB: Doing recovery: scanned up to log sequence number 666 1223804416 InnoDB: Doing recovery: scanned up to log sequence number 666 1229047296 InnoDB: Doing recovery: scanned up to log sequence number 666 1234290176 InnoDB: Doing recovery: scanned up to log sequence number 666 1239533056 InnoDB: Doing recovery: scanned up to log sequence number 666 1244775936 InnoDB: Doing recovery: scanned up to log sequence number 666 1250018816 InnoDB: Doing recovery: scanned up to log sequence number 666 1255261696 InnoDB: Doing recovery: scanned up to log sequence number 666 1260504576 InnoDB: Doing recovery: scanned up to log sequence number 666 1265747456 InnoDB: Doing recovery: scanned up to log sequence number 666 1270990336 InnoDB: Doing recovery: scanned up to log sequence number 666 1276233216 InnoDB: Doing recovery: scanned up to log sequence number 666 1281476096 InnoDB: Doing recovery: scanned up to log sequence number 666 1286718976 InnoDB: Doing recovery: scanned up to log sequence number 666 1291961856 InnoDB: Doing recovery: scanned up to log sequence number 666 1297204736 InnoDB: Doing recovery: scanned up to log sequence number 666 1302447616 InnoDB: Doing recovery: scanned up to log sequence number 666 1307690496 InnoDB: Doing recovery: scanned up to log sequence number 666 1312933376 InnoDB: Doing recovery: scanned up to log sequence number 666 1318176256 InnoDB: Doing recovery: scanned up to log sequence number 666 1323419136 InnoDB: Doing recovery: scanned up to log sequence number 666 1328662016 InnoDB: Doing recovery: scanned up to log sequence number 666 1333904896 InnoDB: Doing recovery: scanned up to log sequence number 666 1339147776 InnoDB: Doing recovery: scanned up to log sequence number 666 1344390656 InnoDB: Doing recovery: scanned up to log sequence number 666 1349633536 InnoDB: Doing recovery: scanned up to log sequence number 666 1354876416 InnoDB: Doing recovery: scanned up to log sequence number 666 1360119296 InnoDB: Doing recovery: scanned up to log sequence number 666 1365362176 InnoDB: Doing recovery: scanned up to log sequence number 666 1370605056 InnoDB: Doing recovery: scanned up to log sequence number 666 1375847936 InnoDB: Doing recovery: scanned up to log sequence number 666 1381090816 InnoDB: Doing recovery: scanned up to log sequence number 666 1386333696 InnoDB: Doing recovery: scanned up to log sequence number 666 1391576576 InnoDB: Doing recovery: scanned up to log sequence number 666 1396819456 InnoDB: Doing recovery: scanned up to log sequence number 666 1402062336 InnoDB: Doing recovery: scanned up to log sequence number 666 1407305216 InnoDB: Doing recovery: scanned up to log sequence number 666 1412548096 InnoDB: Doing recovery: scanned up to log sequence number 666 1417790976 InnoDB: Doing recovery: scanned up to log sequence number 666 1423033856 InnoDB: Doing recovery: scanned up to log sequence number 666 1428276736 InnoDB: Doing recovery: scanned up to log sequence number 666 1433519616 InnoDB: Doing recovery: scanned up to log sequence number 666 1438762496 InnoDB: Doing recovery: scanned up to log sequence number 666 1444005376 InnoDB: Doing recovery: scanned up to log sequence number 666 1449248256 InnoDB: Doing recovery: scanned up to log sequence number 666 1454491136 InnoDB: Doing recovery: scanned up to log sequence number 666 1459734016 InnoDB: Doing recovery: scanned up to log sequence number 666 1464976896 InnoDB: Doing recovery: scanned up to log sequence number 666 1470219776 InnoDB: Doing recovery: scanned up to log sequence number 666 1475462656 InnoDB: Doing recovery: scanned up to log sequence number 666 1480705536 InnoDB: Doing recovery: scanned up to log sequence number 666 1485948416 InnoDB: Doing recovery: scanned up to log sequence number 666 1491191296 InnoDB: Doing recovery: scanned up to log sequence number 666 1496434176 InnoDB: Doing recovery: scanned up to log sequence number 666 1501677056 InnoDB: Doing recovery: scanned up to log sequence number 666 1506919936 InnoDB: Doing recovery: scanned up to log sequence number 666 1512162816 InnoDB: Doing recovery: scanned up to log sequence number 666 1517405696 InnoDB: Doing recovery: scanned up to log sequence number 666 1522648576 InnoDB: Doing recovery: scanned up to log sequence number 666 1527891456 InnoDB: Doing recovery: scanned up to log sequence number 666 1533134336 InnoDB: Doing recovery: scanned up to log sequence number 666 1538377216 InnoDB: Doing recovery: scanned up to log sequence number 666 1543620096 InnoDB: Doing recovery: scanned up to log sequence number 666 1548862976 InnoDB: Doing recovery: scanned up to log sequence number 666 1554105856 InnoDB: Doing recovery: scanned up to log sequence number 666 1559348736 InnoDB: Doing recovery: scanned up to log sequence number 666 1564591616 InnoDB: Doing recovery: scanned up to log sequence number 666 1565023419 InnoDB: 3 transaction(s) which must be rolled back or cleaned up InnoDB: in total 1000522 row operations to undo InnoDB: Trx id counter is 0 33940736 060712 10:24:05 InnoDB: Starting an apply batch of log records to the database... InnoDB: Progress in percents: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 InnoDB: Apply batch completed InnoDB: Starting rollback of uncommitted transactions InnoDB: Rolling back trx with id 0 33940378, 244 rows to undo InnoDB: Rolling back of trx id 0 33940378 completed InnoDB: Rolling back trx with id 0 33939831, 756260 rows to undo InnoDB: Progress in percents: 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 InnoDB: Rolling back of trx id 0 33939831 completed InnoDB: Rolling back trx with id 0 33916775, 244018 rows to undo InnoDB: Progress in percents: 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 InnoDB: Rolling back of trx id 0 33916775 completed InnoDB: Rollback of uncommitted transactions completed InnoDB: In a MySQL replication slave the last master binlog file InnoDB: position 0 200597, file name binlog.000139 InnoDB: Last MySQL binlog file position 0 440906937, file name ./binlog.000100 060712 10:31:54 InnoDB: Flushing modified pages from the buffer pool... 060712 10:32:24 InnoDB: Started; log sequence number 666 1565023419 060712 10:32:24 [ERROR] After InnoDB crash recovery, checking if the binary log './binlog.000100' contains rolled back transactions which must be removed from it... /usr/local/mysql/bin/mysqld: ready for connections. Version: '4.1.18-standard-log' socket: '/tmp/mysql.sock' port: 3306 MySQL Community Edition - Standard (GPL) 060712 10:32:42InnoDB: Assertion failure in thread 51368960 in file dict0load.c line 363 InnoDB: Failing assertion: ut_dulint_cmp(table->id, mach_read_from_8(field)) == 0 InnoDB: We intentionally generate a memory trap. InnoDB: Submit a detailed bug report to http://bugs.mysql.com. InnoDB: If you get repeated assertion failures or crashes, even InnoDB: immediately after the mysqld startup, there may be InnoDB: corruption in the InnoDB tablespace. Please refer to InnoDB: http://dev.mysql.com/doc/mysql/en/Forcing_recovery.html InnoDB: about forcing recovery. InnoDB: Thread 51244032 stopped in file os0sync.c line 501 InnoDB: Thread 50461696 stopped in file ./../include/sync0sync.ic line 111 mysqld got signal 10; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. key_buffer_size=5242880 read_buffer_size=1044480 max_used_connections=33 max_connections=100 threads_connected=15 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_connections = 926319 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. 060712 10:32:44 mysqld restarted [ The db was restarted multiple times. In every case, as soon as the db was put under load it would crash with the 060712 10:32:46 InnoDB: Database was not shut down normally! InnoDB: Starting crash recovery. InnoDB: Reading tablespace information from the .ibd files... InnoDB: Restoring possible half-written data pages from the doublewrite InnoDB: buffer... 060712 10:32:46 InnoDB: Starting log scan based on checkpoint at InnoDB: log sequence number 666 1622525091. InnoDB: Doing recovery: scanned up to log sequence number 666 1627767808 InnoDB: Doing recovery: scanned up to log sequence number 666 1631594677 InnoDB: 1 transaction(s) which must be rolled back or cleaned up InnoDB: in total 1 row operations to undo InnoDB: Trx id counter is 0 33941248 060712 10:32:46 InnoDB: Starting an apply batch of log records to the database... InnoDB: Progress in percents: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 InnoDB: Apply batch completed InnoDB: Starting rollback of uncommitted transactions InnoDB: Rolling back trx with id 0 33940828, 1 rows to undo InnoDB: Rolling back of trx id 0 33940828 completed InnoDB: Rollback of uncommitted transactions completed InnoDB: In a MySQL replication slave the last master binlog file InnoDB: position 0 200597, file name binlog.000139 InnoDB: Last MySQL binlog file position 0 7103588, file name ./binlog.000101 060712 10:32:55 InnoDB: Flushing modified pages from the buffer pool... 060712 10:32:56 InnoDB: Started; log sequence number 666 1631594677 060712 10:32:56 [ERROR] After InnoDB crash recovery, checking if the binary log './binlog.000101' contains rolled back transactions which must be removed from it... /usr/local/mysql/bin/mysqld: ready for connections. Version: '4.1.18-standard-log' socket: '/tmp/mysql.sock' port: 3306 MySQL Community Edition - Standard (GPL) 060712 10:33:44InnoDB: Assertion failure in thread 51250176 in file dict0load.c line 363 InnoDB: Failing assertion: ut_dulint_cmp(table->id, mach_read_from_8(field)) == 0 InnoDB: We intentionally generate a memory trap. InnoDB: Submit a detailed bug report to http://bugs.mysql.com. InnoDB: If you get repeated assertion failures or crashes, even InnoDB: immediately after the mysqld startup, there may be InnoDB: corruption in the InnoDB tablespace. Please refer to InnoDB: http://dev.mysql.com/doc/mysql/en/Forcing_recovery.html InnoDB: about forcing recovery. InnoDB: Thread 50463744 stopped in file os0sync.c line 501 InnoDB: Thread 50461696 stopped in file ./../include/sync0sync.ic line 111 InnoDB: Thread 50459648 stopped in file os0sync.c line 501 mysqld got signal 10; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. key_buffer_size=5242880 read_buffer_size=1044480 max_used_connections=3 max_connections=100 threads_connected=3 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_connections = 926319 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. 060712 10:33:44 mysqld restarted [mysqld was subsequently restarted many times. Each time, when it was put under load it raised the same "ut_dulint_cmp(table->id, mach_read_from_8(field)) == 0" assertion failure]