080129 00:59:43 mysqld started 080129 0:59:44 InnoDB: Started; log sequence number 7 1525962687 080129 1:00:07 [Note] /usr/sbin/mysqld: ready for connections. Version: '5.0.27-standard-log' socket: '/var/lib/mysql/mysql.sock' port: 3306 MySQL Community Edition - Standard (GPL) *** glibc detected *** double free or corruption (!prev): 0x0000002adb0de370 *** mysqld got signal 6; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. key_buffer_size=8388600 read_buffer_size=131072 max_used_connections=63 max_connections=100 threads_connected=21 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_connections = 225791 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. thd=0x2add842360 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... Cannot determine thread, fp=0x45d7bf78, backtrace may not be correct. Stack range sanity check OK, backtrace follows: 0x6f676e697276 Stack trace seems successful - bottom reached Please read http://dev.mysql.com/doc/mysql/en/Using_stack_trace.html and follow instructions on how to resolve the stack trace. Resolved stack trace is much more helpful in diagnosing the problem, so please do resolve it Trying to get some variables. Some pointers may be invalid and cause the dump to abort... thd->query at (nil) is invalid pointer thd->thread_id=24095 The manual page at http://www.mysql.com/doc/en/Crashing.html contains information that should help you find out what is causing the crash. Number of processes running now: 0 080317 16:08:10 mysqld restarted 080317 16:08:11 InnoDB: Database was not shut down normally! InnoDB: Starting crash recovery. InnoDB: Reading tablespace information from the .ibd files... InnoDB: Restoring possible half-written data pages from the doublewrite InnoDB: buffer... 080317 16:08:11 InnoDB: Starting log scan based on checkpoint at InnoDB: log sequence number 8 1724535504. InnoDB: Doing recovery: scanned up to log sequence number 8 1724550249 080317 16:08:11 InnoDB: Starting an apply batch of log records to the database... InnoDB: Progress in percents: 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 InnoDB: Apply batch completed InnoDB: Last MySQL binlog file position 0 102509018, file name /data/mysql-binlog/mysql-binlog.000401 080317 16:08:12 InnoDB: Started; log sequence number 8 1724550249 080317 16:08:12 [Note] Recovering after a crash using /data/mysql-binlog/mysql-binlog 080317 16:08:13 [Note] Starting crash recovery... 080317 16:08:13 [Note] Crash recovery finished. 080317 16:08:13 [Note] /usr/sbin/mysqld: ready for connections. Version: '5.0.27-standard-log' socket: '/var/lib/mysql/mysql.sock' port: 3306 MySQL Community Edition - Standard (GPL) *** glibc detected *** free(): invalid next size (normal): 0x0000000002f20990 *** mysqld got signal 6; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. key_buffer_size=8388600 read_buffer_size=131072 max_used_connections=25 max_connections=100 threads_connected=20 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_connections = 225791 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. thd=0x2adafa8ff0 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... Cannot determine thread, fp=0x45394f78, backtrace may not be correct. Stack range sanity check OK, backtrace follows: 0x6f676e697276 Stack trace seems successful - bottom reached Please read http://dev.mysql.com/doc/mysql/en/Using_stack_trace.html and follow instructions on how to resolve the stack trace. Resolved stack trace is much more helpful in diagnosing the problem, so please do resolve it Trying to get some variables. Some pointers may be invalid and cause the dump to abort... thd->query at (nil) is invalid pointer thd->thread_id=13 The manual page at http://www.mysql.com/doc/en/Crashing.html contains information that should help you find out what is causing the crash. mysqld: my_new.cc:51: int __cxa_pure_virtual(): Assertion `"Pure virtual method called." == "Aborted"' failed. Number of processes running now: 0 080317 17:19:33 mysqld restarted 080317 17:19:34 InnoDB: Database was not shut down normally! InnoDB: Starting crash recovery. InnoDB: Reading tablespace information from the .ibd files... InnoDB: Restoring possible half-written data pages from the doublewrite InnoDB: buffer... 080317 17:19:34 InnoDB: Starting log scan based on checkpoint at InnoDB: log sequence number 8 1728923820. InnoDB: Doing recovery: scanned up to log sequence number 8 1728930659 InnoDB: 1 transaction(s) which must be rolled back or cleaned up InnoDB: in total 2 row operations to undo InnoDB: Trx id counter is 0 285599232 080317 17:19:34 InnoDB: Starting an apply batch of log records to the database... InnoDB: Progress in percents: 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 InnoDB: Apply batch completed InnoDB: Last MySQL binlog file position 0 3304052, file name /data/mysql-binlog/mysql-binlog.000402 InnoDB: Starting in background the rollback of uncommitted transactions 080317 17:19:35 InnoDB: Rolling back trx with id 0 285597817, 2 rows to undo 080317 17:19:35 InnoDB: Started; log sequence number 8 1728930659 080317 17:19:35 [Note] Recovering after a crash using /data/mysql-binlog/mysql-binlog InnoDB: Rolling back of trx id 0 285597817 completed 080317 17:19:35 InnoDB: Rollback of non-prepared transactions completed 080317 17:19:35 [Note] Starting crash recovery... 080317 17:19:35 [Note] Crash recovery finished. 080317 17:19:35 [Note] /usr/sbin/mysqld: ready for connections. Version: '5.0.27-standard-log' socket: '/var/lib/mysql/mysql.sock' port: 3306 MySQL Community Edition - Standard (GPL) *** glibc detected *** double free or corruption (!prev): 0x0000000003328a50 *** mysqld got signal 6; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. key_buffer_size=8388600 read_buffer_size=131072 max_used_connections=50 max_connections=100 threads_connected=11 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_connections = 225791 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. thd=0x2adb56b4a0 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... Cannot determine thread, fp=0x455ddf78, backtrace may not be correct. Stack range sanity check OK, backtrace follows: 0x6f676e697276 Stack trace seems successful - bottom reached Please read http://dev.mysql.com/doc/mysql/en/Using_stack_trace.html and follow instructions on how to resolve the stack trace. Resolved stack trace is much more helpful in diagnosing the problem, so please do resolve it Trying to get some variables. Some pointers may be invalid and cause the dump to abort... thd->query at (nil) is invalid pointer thd->thread_id=261 The manual page at http://www.mysql.com/doc/en/Crashing.html contains information that should help you find out what is causing the crash. InnoDB: Warning: a long semaphore wait: --Thread 1161120096 has waited at trx0trx.c line 213 for 242.00 seconds the semaphore: Mutex at 0x2a95b6b2b8 created file srv0srv.c line 872, lock var 1 waiters flag 1 InnoDB: Warning: a long semaphore wait: --Thread 1157658976 has waited at row0mysql.c line 2233 for 241.00 seconds the semaphore: Mutex at 0x2a95b6b2b8 created file srv0srv.c line 872, lock var 1 waiters flag 1 InnoDB: Warning: a long semaphore wait: --Thread 1136679264 has waited at srv0srv.c line 1952 for 241.00 seconds the semaphore: Mutex at 0x2a95b6b2b8 created file srv0srv.c line 872, lock var 1 waiters flag 1 InnoDB: ###### Starts InnoDB Monitor for 30 secs to print diagnostic info: InnoDB: Pending preads 0, pwrites 0 InnoDB: ###### Diagnostic info printed to the standard error stream InnoDB: Warning: a long semaphore wait: --Thread 1161120096 has waited at trx0trx.c line 213 for 274.00 seconds the semaphore: Mutex at 0x2a95b6b2b8 created file srv0srv.c line 872, lock var 1 waiters flag 1 InnoDB: Warning: a long semaphore wait: --Thread 1157658976 has waited at row0mysql.c line 2233 for 273.00 seconds the semaphore: Mutex at 0x2a95b6b2b8 created file srv0srv.c line 872, lock var 1 waiters flag 1 InnoDB: Warning: a long semaphore wait: --Thread 1136679264 has waited at srv0srv.c line 1952 for 273.00 seconds the semaphore: Mutex at 0x2a95b6b2b8 created file srv0srv.c line 872, lock var 1 waiters flag 1 InnoDB: Warning: a long semaphore wait: --Thread 1169906016 has waited at trx0trx.c line 213 for 265.00 seconds the semaphore: Mutex at 0x2a95b6b2b8 created file srv0srv.c line 872, lock var 1 waiters flag 1 InnoDB: Warning: a long semaphore wait: --Thread 1169639776 has waited at trx0trx.c line 213 for 264.00 seconds the semaphore: Mutex at 0x2a95b6b2b8 created file srv0srv.c line 872, lock var 1 waiters flag 1 InnoDB: Warning: a long semaphore wait: --Thread 1159788896 has waited at trx0trx.c line 715 for 263.00 seconds the semaphore: Mutex at 0x2a95b6b2b8 created file srv0srv.c line 872, lock var 1 waiters flag 1 InnoDB: ###### Starts InnoDB Monitor for 30 secs to print diagnostic info: InnoDB: Pending preads 0, pwrites 0 InnoDB: ###### Diagnostic info printed to the standard error stream InnoDB: Warning: a long semaphore wait: --Thread 1161120096 has waited at trx0trx.c line 213 for 306.00 seconds the semaphore: Mutex at 0x2a95b6b2b8 created file srv0srv.c line 872, lock var 1 waiters flag 1 InnoDB: Warning: a long semaphore wait: --Thread 1157658976 has waited at row0mysql.c line 2233 for 305.00 seconds the semaphore: Mutex at 0x2a95b6b2b8 created file srv0srv.c line 872, lock var 1 waiters flag 1 InnoDB: Warning: a long semaphore wait: --Thread 1136679264 has waited at srv0srv.c line 1952 for 305.00 seconds the semaphore: Mutex at 0x2a95b6b2b8 created file srv0srv.c line 872, lock var 1 waiters flag 1 InnoDB: Warning: a long semaphore wait: --Thread 1169906016 has waited at trx0trx.c line 213 for 297.00 seconds the semaphore: Mutex at 0x2a95b6b2b8 created file srv0srv.c line 872, lock var 1 waiters flag 1 InnoDB: Warning: a long semaphore wait: --Thread 1169639776 has waited at trx0trx.c line 213 for 296.00 seconds the semaphore: Mutex at 0x2a95b6b2b8 created file srv0srv.c line 872, lock var 1 waiters flag 1 InnoDB: Warning: a long semaphore wait: --Thread 1159788896 has waited at trx0trx.c line 715 for 295.00 seconds the semaphore: Mutex at 0x2a95b6b2b8 created file srv0srv.c line 872, lock var 1 waiters flag 1 InnoDB: ###### Starts InnoDB Monitor for 30 secs to print diagnostic info: InnoDB: Pending preads 0, pwrites 0 InnoDB: ###### Diagnostic info printed to the standard error stream InnoDB: Warning: a long semaphore wait: --Thread 1161120096 has waited at trx0trx.c line 213 for 338.00 seconds the semaphore: Mutex at 0x2a95b6b2b8 created file srv0srv.c line 872, lock var 1 waiters flag 1 InnoDB: Warning: a long semaphore wait: --Thread 1157658976 has waited at row0mysql.c line 2233 for 337.00 seconds the semaphore: Mutex at 0x2a95b6b2b8 created file srv0srv.c line 872, lock var 1 waiters flag 1 InnoDB: Warning: a long semaphore wait: --Thread 1136679264 has waited at srv0srv.c line 1952 for 337.00 seconds the semaphore: Mutex at 0x2a95b6b2b8 created file srv0srv.c line 872, lock var 1 waiters flag 1 InnoDB: Warning: a long semaphore wait: --Thread 1169906016 has waited at trx0trx.c line 213 for 329.00 seconds the semaphore: Mutex at 0x2a95b6b2b8 created file srv0srv.c line 872, lock var 1 waiters flag 1 InnoDB: Warning: a long semaphore wait: --Thread 1169639776 has waited at trx0trx.c line 213 for 328.00 seconds the semaphore: Mutex at 0x2a95b6b2b8 created file srv0srv.c line 872, lock var 1 waiters flag 1 InnoDB: Warning: a long semaphore wait: --Thread 1159788896 has waited at trx0trx.c line 715 for 327.00 seconds the semaphore: Mutex at 0x2a95b6b2b8 created file srv0srv.c line 872, lock var 1 waiters flag 1 InnoDB: ###### Starts InnoDB Monitor for 30 secs to print diagnostic info: InnoDB: Pending preads 0, pwrites 0 InnoDB: ###### Diagnostic info printed to the standard error stream InnoDB: Warning: a long semaphore wait: --Thread 1161120096 has waited at trx0trx.c line 213 for 370.00 seconds the semaphore: Mutex at 0x2a95b6b2b8 created file srv0srv.c line 872, lock var 1 waiters flag 1 InnoDB: Warning: a long semaphore wait: --Thread 1157658976 has waited at row0mysql.c line 2233 for 369.00 seconds the semaphore: Mutex at 0x2a95b6b2b8 created file srv0srv.c line 872, lock var 1 waiters flag 1 InnoDB: Warning: a long semaphore wait: --Thread 1136679264 has waited at srv0srv.c line 1952 for 369.00 seconds the semaphore: Mutex at 0x2a95b6b2b8 created file srv0srv.c line 872, lock var 1 waiters flag 1 InnoDB: Warning: a long semaphore wait: --Thread 1169906016 has waited at trx0trx.c line 213 for 361.00 seconds the semaphore: Mutex at 0x2a95b6b2b8 created file srv0srv.c line 872, lock var 1 waiters flag 1 InnoDB: Warning: a long semaphore wait: --Thread 1169639776 has waited at trx0trx.c line 213 for 360.00 seconds the semaphore: Mutex at 0x2a95b6b2b8 created file srv0srv.c line 872, lock var 1 waiters flag 1 InnoDB: Warning: a long semaphore wait: --Thread 1159788896 has waited at trx0trx.c line 715 for 359.00 seconds the semaphore: Mutex at 0x2a95b6b2b8 created file srv0srv.c line 872, lock var 1 waiters flag 1 InnoDB: ###### Starts InnoDB Monitor for 30 secs to print diagnostic info: InnoDB: Pending preads 0, pwrites 0 InnoDB: ###### Diagnostic info printed to the standard error stream InnoDB: Warning: a long semaphore wait: --Thread 1161120096 has waited at trx0trx.c line 213 for 402.00 seconds the semaphore: Mutex at 0x2a95b6b2b8 created file srv0srv.c line 872, lock var 1 waiters flag 1 InnoDB: Warning: a long semaphore wait: --Thread 1157658976 has waited at row0mysql.c line 2233 for 401.00 seconds the semaphore: Mutex at 0x2a95b6b2b8 created file srv0srv.c line 872, lock var 1 waiters flag 1 InnoDB: Warning: a long semaphore wait: --Thread 1136679264 has waited at srv0srv.c line 1952 for 401.00 seconds the semaphore: Mutex at 0x2a95b6b2b8 created file srv0srv.c line 872, lock var 1 waiters flag 1 InnoDB: Warning: a long semaphore wait: --Thread 1169906016 has waited at trx0trx.c line 213 for 393.00 seconds the semaphore: Mutex at 0x2a95b6b2b8 created file srv0srv.c line 872, lock var 1 waiters flag 1 InnoDB: Warning: a long semaphore wait: --Thread 1169639776 has waited at trx0trx.c line 213 for 392.00 seconds the semaphore: Mutex at 0x2a95b6b2b8 created file srv0srv.c line 872, lock var 1 waiters flag 1 InnoDB: Warning: a long semaphore wait: --Thread 1159788896 has waited at trx0trx.c line 715 for 391.00 seconds the semaphore: Mutex at 0x2a95b6b2b8 created file srv0srv.c line 872, lock var 1 waiters flag 1 InnoDB: ###### Starts InnoDB Monitor for 30 secs to print diagnostic info: InnoDB: Pending preads 0, pwrites 0 InnoDB: ###### Diagnostic info printed to the standard error stream InnoDB: Warning: a long semaphore wait: --Thread 1161120096 has waited at trx0trx.c line 213 for 434.00 seconds the semaphore: Mutex at 0x2a95b6b2b8 created file srv0srv.c line 872, lock var 1 waiters flag 1 InnoDB: Warning: a long semaphore wait: --Thread 1157658976 has waited at row0mysql.c line 2233 for 433.00 seconds the semaphore: Mutex at 0x2a95b6b2b8 created file srv0srv.c line 872, lock var 1 waiters flag 1 InnoDB: Warning: a long semaphore wait: --Thread 1136679264 has waited at srv0srv.c line 1952 for 433.00 seconds the semaphore: Mutex at 0x2a95b6b2b8 created file srv0srv.c line 872, lock var 1 waiters flag 1 InnoDB: Warning: a long semaphore wait: --Thread 1169906016 has waited at trx0trx.c line 213 for 425.00 seconds the semaphore: Mutex at 0x2a95b6b2b8 created file srv0srv.c line 872, lock var 1 waiters flag 1 InnoDB: Warning: a long semaphore wait: --Thread 1169639776 has waited at trx0trx.c line 213 for 424.00 seconds the semaphore: Mutex at 0x2a95b6b2b8 created file srv0srv.c line 872, lock var 1 waiters flag 1 InnoDB: Warning: a long semaphore wait: --Thread 1159788896 has waited at trx0trx.c line 715 for 423.00 seconds the semaphore: Mutex at 0x2a95b6b2b8 created file srv0srv.c line 872, lock var 1 waiters flag 1 InnoDB: ###### Starts InnoDB Monitor for 30 secs to print diagnostic info: InnoDB: Pending preads 0, pwrites 0 InnoDB: ###### Diagnostic info printed to the standard error stream InnoDB: Warning: a long semaphore wait: --Thread 1161120096 has waited at trx0trx.c line 213 for 466.00 seconds the semaphore: Mutex at 0x2a95b6b2b8 created file srv0srv.c line 872, lock var 1 waiters flag 1 InnoDB: Warning: a long semaphore wait: --Thread 1157658976 has waited at row0mysql.c line 2233 for 465.00 seconds the semaphore: Mutex at 0x2a95b6b2b8 created file srv0srv.c line 872, lock var 1 waiters flag 1 InnoDB: Warning: a long semaphore wait: --Thread 1136679264 has waited at srv0srv.c line 1952 for 465.00 seconds the semaphore: Mutex at 0x2a95b6b2b8 created file srv0srv.c line 872, lock var 1 waiters flag 1 InnoDB: Warning: a long semaphore wait: --Thread 1169906016 has waited at trx0trx.c line 213 for 457.00 seconds the semaphore: Mutex at 0x2a95b6b2b8 created file srv0srv.c line 872, lock var 1 waiters flag 1 InnoDB: Warning: a long semaphore wait: --Thread 1169639776 has waited at trx0trx.c line 213 for 456.00 seconds the semaphore: Mutex at 0x2a95b6b2b8 created file srv0srv.c line 872, lock var 1 waiters flag 1 InnoDB: Warning: a long semaphore wait: --Thread 1159788896 has waited at trx0trx.c line 715 for 455.00 seconds the semaphore: Mutex at 0x2a95b6b2b8 created file srv0srv.c line 872, lock var 1 waiters flag 1 InnoDB: ###### Starts InnoDB Monitor for 30 secs to print diagnostic info: InnoDB: Pending preads 0, pwrites 0 InnoDB: ###### Diagnostic info printed to the standard error stream InnoDB: Warning: a long semaphore wait: --Thread 1161120096 has waited at trx0trx.c line 213 for 498.00 seconds the semaphore: Mutex at 0x2a95b6b2b8 created file srv0srv.c line 872, lock var 1 waiters flag 1 InnoDB: Warning: a long semaphore wait: --Thread 1157658976 has waited at row0mysql.c line 2233 for 497.00 seconds the semaphore: Mutex at 0x2a95b6b2b8 created file srv0srv.c line 872, lock var 1 waiters flag 1 InnoDB: Warning: a long semaphore wait: --Thread 1136679264 has waited at srv0srv.c line 1952 for 497.00 seconds the semaphore: Mutex at 0x2a95b6b2b8 created file srv0srv.c line 872, lock var 1 waiters flag 1 InnoDB: Warning: a long semaphore wait: --Thread 1169906016 has waited at trx0trx.c line 213 for 489.00 seconds the semaphore: Mutex at 0x2a95b6b2b8 created file srv0srv.c line 872, lock var 1 waiters flag 1 InnoDB: Warning: a long semaphore wait: --Thread 1169639776 has waited at trx0trx.c line 213 for 488.00 seconds the semaphore: Mutex at 0x2a95b6b2b8 created file srv0srv.c line 872, lock var 1 waiters flag 1 InnoDB: Warning: a long semaphore wait: --Thread 1159788896 has waited at trx0trx.c line 715 for 487.00 seconds the semaphore: Mutex at 0x2a95b6b2b8 created file srv0srv.c line 872, lock var 1 waiters flag 1 InnoDB: ###### Starts InnoDB Monitor for 30 secs to print diagnostic info: InnoDB: Pending preads 0, pwrites 0 InnoDB: ###### Diagnostic info printed to the standard error stream InnoDB: Warning: a long semaphore wait: --Thread 1161120096 has waited at trx0trx.c line 213 for 530.00 seconds the semaphore: Mutex at 0x2a95b6b2b8 created file srv0srv.c line 872, lock var 1 waiters flag 1 InnoDB: Warning: a long semaphore wait: --Thread 1157658976 has waited at row0mysql.c line 2233 for 529.00 seconds the semaphore: Mutex at 0x2a95b6b2b8 created file srv0srv.c line 872, lock var 1 waiters flag 1 InnoDB: Warning: a long semaphore wait: --Thread 1136679264 has waited at srv0srv.c line 1952 for 529.00 seconds the semaphore: Mutex at 0x2a95b6b2b8 created file srv0srv.c line 872, lock var 1 waiters flag 1 InnoDB: Warning: a long semaphore wait: --Thread 1169906016 has waited at trx0trx.c line 213 for 521.00 seconds the semaphore: Mutex at 0x2a95b6b2b8 created file srv0srv.c line 872, lock var 1 waiters flag 1 InnoDB: Warning: a long semaphore wait: --Thread 1169639776 has waited at trx0trx.c line 213 for 520.00 seconds the semaphore: Mutex at 0x2a95b6b2b8 created file srv0srv.c line 872, lock var 1 waiters flag 1 InnoDB: Warning: a long semaphore wait: --Thread 1159788896 has waited at trx0trx.c line 715 for 519.00 seconds the semaphore: Mutex at 0x2a95b6b2b8 created file srv0srv.c line 872, lock var 1 waiters flag 1 InnoDB: ###### Starts InnoDB Monitor for 30 secs to print diagnostic info: InnoDB: Pending preads 0, pwrites 0 InnoDB: ###### Diagnostic info printed to the standard error stream InnoDB: Warning: a long semaphore wait: --Thread 1161120096 has waited at trx0trx.c line 213 for 562.00 seconds the semaphore: Mutex at 0x2a95b6b2b8 created file srv0srv.c line 872, lock var 1 waiters flag 1 InnoDB: Warning: a long semaphore wait: --Thread 1157658976 has waited at row0mysql.c line 2233 for 561.00 seconds the semaphore: Mutex at 0x2a95b6b2b8 created file srv0srv.c line 872, lock var 1 waiters flag 1 InnoDB: Warning: a long semaphore wait: --Thread 1136679264 has waited at srv0srv.c line 1952 for 561.00 seconds the semaphore: Mutex at 0x2a95b6b2b8 created file srv0srv.c line 872, lock var 1 waiters flag 1 InnoDB: Warning: a long semaphore wait: --Thread 1169906016 has waited at trx0trx.c line 213 for 553.00 seconds the semaphore: Mutex at 0x2a95b6b2b8 created file srv0srv.c line 872, lock var 1 waiters flag 1 InnoDB: Warning: a long semaphore wait: --Thread 1169639776 has waited at trx0trx.c line 213 for 552.00 seconds the semaphore: Mutex at 0x2a95b6b2b8 created file srv0srv.c line 872, lock var 1 waiters flag 1 InnoDB: Warning: a long semaphore wait: --Thread 1159788896 has waited at trx0trx.c line 715 for 551.00 seconds the semaphore: Mutex at 0x2a95b6b2b8 created file srv0srv.c line 872, lock var 1 waiters flag 1 InnoDB: ###### Starts InnoDB Monitor for 30 secs to print diagnostic info: InnoDB: Pending preads 0, pwrites 0 InnoDB: ###### Diagnostic info printed to the standard error stream InnoDB: Warning: a long semaphore wait: --Thread 1161120096 has waited at trx0trx.c line 213 for 594.00 seconds the semaphore: Mutex at 0x2a95b6b2b8 created file srv0srv.c line 872, lock var 1 waiters flag 1 InnoDB: Warning: a long semaphore wait: --Thread 1157658976 has waited at row0mysql.c line 2233 for 593.00 seconds the semaphore: Mutex at 0x2a95b6b2b8 created file srv0srv.c line 872, lock var 1 waiters flag 1 InnoDB: Warning: a long semaphore wait: --Thread 1136679264 has waited at srv0srv.c line 1952 for 593.00 seconds the semaphore: Mutex at 0x2a95b6b2b8 created file srv0srv.c line 872, lock var 1 waiters flag 1 InnoDB: Warning: a long semaphore wait: --Thread 1169906016 has waited at trx0trx.c line 213 for 585.00 seconds the semaphore: Mutex at 0x2a95b6b2b8 created file srv0srv.c line 872, lock var 1 waiters flag 1 InnoDB: Warning: a long semaphore wait: --Thread 1169639776 has waited at trx0trx.c line 213 for 584.00 seconds the semaphore: Mutex at 0x2a95b6b2b8 created file srv0srv.c line 872, lock var 1 waiters flag 1 InnoDB: Warning: a long semaphore wait: --Thread 1159788896 has waited at trx0trx.c line 715 for 583.00 seconds the semaphore: Mutex at 0x2a95b6b2b8 created file srv0srv.c line 872, lock var 1 waiters flag 1 InnoDB: ###### Starts InnoDB Monitor for 30 secs to print diagnostic info: InnoDB: Pending preads 0, pwrites 0 InnoDB: ###### Diagnostic info printed to the standard error stream InnoDB: Warning: a long semaphore wait: --Thread 1161120096 has waited at trx0trx.c line 213 for 626.00 seconds the semaphore: Mutex at 0x2a95b6b2b8 created file srv0srv.c line 872, lock var 1 waiters flag 1 InnoDB: Warning: a long semaphore wait: --Thread 1157658976 has waited at row0mysql.c line 2233 for 625.00 seconds the semaphore: Mutex at 0x2a95b6b2b8 created file srv0srv.c line 872, lock var 1 waiters flag 1 InnoDB: Warning: a long semaphore wait: --Thread 1136679264 has waited at srv0srv.c line 1952 for 625.00 seconds the semaphore: Mutex at 0x2a95b6b2b8 created file srv0srv.c line 872, lock var 1 waiters flag 1 InnoDB: Warning: a long semaphore wait: --Thread 1169906016 has waited at trx0trx.c line 213 for 617.00 seconds the semaphore: Mutex at 0x2a95b6b2b8 created file srv0srv.c line 872, lock var 1 waiters flag 1 InnoDB: Warning: a long semaphore wait: --Thread 1169639776 has waited at trx0trx.c line 213 for 616.00 seconds the semaphore: Mutex at 0x2a95b6b2b8 created file srv0srv.c line 872, lock var 1 waiters flag 1 InnoDB: Warning: a long semaphore wait: --Thread 1159788896 has waited at trx0trx.c line 715 for 615.00 seconds the semaphore: Mutex at 0x2a95b6b2b8 created file srv0srv.c line 872, lock var 1 waiters flag 1 InnoDB: ###### Starts InnoDB Monitor for 30 secs to print diagnostic info: InnoDB: Pending preads 0, pwrites 0 InnoDB: ###### Diagnostic info printed to the standard error stream InnoDB: Warning: a long semaphore wait: --Thread 1161120096 has waited at trx0trx.c line 213 for 658.00 seconds the semaphore: Mutex at 0x2a95b6b2b8 created file srv0srv.c line 872, lock var 1 waiters flag 1 InnoDB: Warning: a long semaphore wait: --Thread 1157658976 has waited at row0mysql.c line 2233 for 657.00 seconds the semaphore: Mutex at 0x2a95b6b2b8 created file srv0srv.c line 872, lock var 1 waiters flag 1 InnoDB: Warning: a long semaphore wait: --Thread 1136679264 has waited at srv0srv.c line 1952 for 657.00 seconds the semaphore: Mutex at 0x2a95b6b2b8 created file srv0srv.c line 872, lock var 1 waiters flag 1 InnoDB: Warning: a long semaphore wait: --Thread 1169906016 has waited at trx0trx.c line 213 for 649.00 seconds the semaphore: Mutex at 0x2a95b6b2b8 created file srv0srv.c line 872, lock var 1 waiters flag 1 InnoDB: Warning: a long semaphore wait: --Thread 1169639776 has waited at trx0trx.c line 213 for 648.00 seconds the semaphore: Mutex at 0x2a95b6b2b8 created file srv0srv.c line 872, lock var 1 waiters flag 1 InnoDB: Warning: a long semaphore wait: --Thread 1159788896 has waited at trx0trx.c line 715 for 647.00 seconds the semaphore: Mutex at 0x2a95b6b2b8 created file srv0srv.c line 872, lock var 1 waiters flag 1 InnoDB: ###### Starts InnoDB Monitor for 30 secs to print diagnostic info: InnoDB: Pending preads 0, pwrites 0 InnoDB: ###### Diagnostic info printed to the standard error stream InnoDB: Warning: a long semaphore wait: --Thread 1161120096 has waited at trx0trx.c line 213 for 690.00 seconds the semaphore: Mutex at 0x2a95b6b2b8 created file srv0srv.c line 872, lock var 1 waiters flag 1 InnoDB: Warning: a long semaphore wait: --Thread 1157658976 has waited at row0mysql.c line 2233 for 689.00 seconds the semaphore: Mutex at 0x2a95b6b2b8 created file srv0srv.c line 872, lock var 1 waiters flag 1 InnoDB: Warning: a long semaphore wait: --Thread 1136679264 has waited at srv0srv.c line 1952 for 689.00 seconds the semaphore: Mutex at 0x2a95b6b2b8 created file srv0srv.c line 872, lock var 1 waiters flag 1 InnoDB: Warning: a long semaphore wait: --Thread 1169906016 has waited at trx0trx.c line 213 for 681.00 seconds the semaphore: Mutex at 0x2a95b6b2b8 created file srv0srv.c line 872, lock var 1 waiters flag 1 InnoDB: Warning: a long semaphore wait: --Thread 1169639776 has waited at trx0trx.c line 213 for 680.00 seconds the semaphore: Mutex at 0x2a95b6b2b8 created file srv0srv.c line 872, lock var 1 waiters flag 1 InnoDB: Warning: a long semaphore wait: --Thread 1159788896 has waited at trx0trx.c line 715 for 679.00 seconds the semaphore: Mutex at 0x2a95b6b2b8 created file srv0srv.c line 872, lock var 1 waiters flag 1 InnoDB: ###### Starts InnoDB Monitor for 30 secs to print diagnostic info: InnoDB: Pending preads 0, pwrites 0 InnoDB: ###### Diagnostic info printed to the standard error stream InnoDB: Warning: a long semaphore wait: --Thread 1161120096 has waited at trx0trx.c line 213 for 722.00 seconds the semaphore: Mutex at 0x2a95b6b2b8 created file srv0srv.c line 872, lock var 1 waiters flag 1 InnoDB: Warning: a long semaphore wait: --Thread 1157658976 has waited at row0mysql.c line 2233 for 721.00 seconds the semaphore: Mutex at 0x2a95b6b2b8 created file srv0srv.c line 872, lock var 1 waiters flag 1 InnoDB: Warning: a long semaphore wait: --Thread 1136679264 has waited at srv0srv.c line 1952 for 721.00 seconds the semaphore: Mutex at 0x2a95b6b2b8 created file srv0srv.c line 872, lock var 1 waiters flag 1 InnoDB: Warning: a long semaphore wait: --Thread 1169906016 has waited at trx0trx.c line 213 for 713.00 seconds the semaphore: Mutex at 0x2a95b6b2b8 created file srv0srv.c line 872, lock var 1 waiters flag 1 InnoDB: Warning: a long semaphore wait: --Thread 1169639776 has waited at trx0trx.c line 213 for 712.00 seconds the semaphore: Mutex at 0x2a95b6b2b8 created file srv0srv.c line 872, lock var 1 waiters flag 1 InnoDB: Warning: a long semaphore wait: --Thread 1159788896 has waited at trx0trx.c line 715 for 711.00 seconds the semaphore: Mutex at 0x2a95b6b2b8 created file srv0srv.c line 872, lock var 1 waiters flag 1 InnoDB: ###### Starts InnoDB Monitor for 30 secs to print diagnostic info: InnoDB: Pending preads 0, pwrites 0 InnoDB: ###### Diagnostic info printed to the standard error stream InnoDB: Warning: a long semaphore wait: --Thread 1161120096 has waited at trx0trx.c line 213 for 754.00 seconds the semaphore: Mutex at 0x2a95b6b2b8 created file srv0srv.c line 872, lock var 1 waiters flag 1 InnoDB: Warning: a long semaphore wait: --Thread 1157658976 has waited at row0mysql.c line 2233 for 753.00 seconds the semaphore: Mutex at 0x2a95b6b2b8 created file srv0srv.c line 872, lock var 1 waiters flag 1 InnoDB: Warning: a long semaphore wait: --Thread 1136679264 has waited at srv0srv.c line 1952 for 753.00 seconds the semaphore: Mutex at 0x2a95b6b2b8 created file srv0srv.c line 872, lock var 1 waiters flag 1 InnoDB: Warning: a long semaphore wait: --Thread 1169906016 has waited at trx0trx.c line 213 for 745.00 seconds the semaphore: Mutex at 0x2a95b6b2b8 created file srv0srv.c line 872, lock var 1 waiters flag 1 InnoDB: Warning: a long semaphore wait: --Thread 1169639776 has waited at trx0trx.c line 213 for 744.00 seconds the semaphore: Mutex at 0x2a95b6b2b8 created file srv0srv.c line 872, lock var 1 waiters flag 1 InnoDB: Warning: a long semaphore wait: --Thread 1159788896 has waited at trx0trx.c line 715 for 743.00 seconds the semaphore: Mutex at 0x2a95b6b2b8 created file srv0srv.c line 872, lock var 1 waiters flag 1 InnoDB: ###### Starts InnoDB Monitor for 30 secs to print diagnostic info: InnoDB: Pending preads 0, pwrites 0 InnoDB: ###### Diagnostic info printed to the standard error stream InnoDB: Warning: a long semaphore wait: --Thread 1161120096 has waited at trx0trx.c line 213 for 786.00 seconds the semaphore: Mutex at 0x2a95b6b2b8 created file srv0srv.c line 872, lock var 1 waiters flag 1 InnoDB: Warning: a long semaphore wait: --Thread 1157658976 has waited at row0mysql.c line 2233 for 785.00 seconds the semaphore: Mutex at 0x2a95b6b2b8 created file srv0srv.c line 872, lock var 1 waiters flag 1 InnoDB: Warning: a long semaphore wait: --Thread 1136679264 has waited at srv0srv.c line 1952 for 785.00 seconds the semaphore: Mutex at 0x2a95b6b2b8 created file srv0srv.c line 872, lock var 1 waiters flag 1 InnoDB: Warning: a long semaphore wait: --Thread 1169906016 has waited at trx0trx.c line 213 for 777.00 seconds the semaphore: Mutex at 0x2a95b6b2b8 created file srv0srv.c line 872, lock var 1 waiters flag 1 InnoDB: Warning: a long semaphore wait: --Thread 1169639776 has waited at trx0trx.c line 213 for 776.00 seconds the semaphore: Mutex at 0x2a95b6b2b8 created file srv0srv.c line 872, lock var 1 waiters flag 1 InnoDB: Warning: a long semaphore wait: --Thread 1159788896 has waited at trx0trx.c line 715 for 775.00 seconds the semaphore: Mutex at 0x2a95b6b2b8 created file srv0srv.c line 872, lock var 1 waiters flag 1 InnoDB: ###### Starts InnoDB Monitor for 30 secs to print diagnostic info: InnoDB: Pending preads 0, pwrites 0 InnoDB: ###### Diagnostic info printed to the standard error stream InnoDB: Error: semaphore wait has lasted > 600 seconds InnoDB: We intentionally crash the server, because it appears to be hung. 080318 6:14:19InnoDB: Assertion failure in thread 1147169120 in file srv0srv.c line 2093 InnoDB: We intentionally generate a memory trap. InnoDB: Submit a detailed bug report to http://bugs.mysql.com. InnoDB: If you get repeated assertion failures or crashes, even InnoDB: immediately after the mysqld startup, there may be InnoDB: corruption in the InnoDB tablespace. Please refer to InnoDB: http://dev.mysql.com/doc/refman/5.0/en/forcing-recovery.html InnoDB: about forcing recovery. Fatal signal 11 while backtracing InnoDB: Thread 1158191456 stopped in file ha_innodb.cc line 2019 080318 6:40:29 [Note] /usr/sbin/mysqld: Normal shutdown InnoDB: Thread 1158191456 stopped in file ./../include/sync0sync.ic line 111 080318 06:49:06 mysqld started 080318 6:49:07 InnoDB: Database was not shut down normally! InnoDB: Starting crash recovery. InnoDB: Reading tablespace information from the .ibd files... InnoDB: Restoring possible half-written data pages from the doublewrite InnoDB: buffer... 080318 6:49:07 InnoDB: Starting log scan based on checkpoint at InnoDB: log sequence number 8 1772857380. InnoDB: Doing recovery: scanned up to log sequence number 8 1772867929 InnoDB: 1 transaction(s) which must be rolled back or cleaned up InnoDB: in total 1 row operations to undo InnoDB: Trx id counter is 0 285853952 080318 6:49:07 InnoDB: Starting an apply batch of log records to the database... InnoDB: Progress in percents: 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 InnoDB: Apply batch completed InnoDB: Last MySQL binlog file position 0 32211514, file name /data/mysql-binlog/mysql-binlog.000403 InnoDB: Starting in background the rollback of uncommitted transactions 080318 6:49:08 InnoDB: Rolling back trx with id 0 285852839, 1 rows to undo 080318 6:49:08 InnoDB: Started; log sequence number 8 1772867929 080318 6:49:08 [Note] Recovering after a crash using /data/mysql-binlog/mysql-binlog InnoDB: Rolling back of trx id 0 285852839 completed 080318 6:49:08 InnoDB: Rollback of non-prepared transactions completed 080318 6:49:08 [Note] Starting crash recovery... 080318 6:49:08 [Note] Crash recovery finished. 080318 6:49:08 [Note] /usr/sbin/mysqld: ready for connections. Version: '5.0.27-standard-log' socket: '/var/lib/mysql/mysql.sock' port: 3306 MySQL Community Edition - Standard (GPL) mysqld got signal 11; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. key_buffer_size=8388600 read_buffer_size=131072 max_used_connections=60 max_connections=100 threads_connected=19 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_connections = 225791 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. thd=0x2f9c4f0 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... Cannot determine thread, fp=0x45af1f78, backtrace may not be correct. Stack range sanity check OK, backtrace follows: 0x6f676e697276 New value of fp=0x2f9c4f0 failed sanity check, terminating stack trace! Please read http://dev.mysql.com/doc/mysql/en/Using_stack_trace.html and follow instructions on how to resolve the stack trace. Resolved stack trace is much more helpful in diagnosing the problem, so please do resolve it Trying to get some variables. Some pointers may be invalid and cause the dump to abort... thd->query at (nil) is invalid pointer thd->thread_id=83 The manual page at http://www.mysql.com/doc/en/Crashing.html contains information that should help you find out what is causing the crash. 080318 9:25:54 [Note] /usr/sbin/mysqld: Normal shutdown Number of processes running now: 0 080318 09:26:45 mysqld restarted 080318 9:26:45 InnoDB: Database was not shut down normally! InnoDB: Starting crash recovery. InnoDB: Reading tablespace information from the .ibd files... InnoDB: Restoring possible half-written data pages from the doublewrite InnoDB: buffer... 080318 9:26:45 InnoDB: Starting log scan based on checkpoint at InnoDB: log sequence number 8 1783238603. InnoDB: Doing recovery: scanned up to log sequence number 8 1783238603 InnoDB: Last MySQL binlog file position 0 7922492, file name /data/mysql-binlog/mysql-binlog.000404 080318 9:26:46 InnoDB: Started; log sequence number 8 1783238603 080318 9:26:46 [Note] Recovering after a crash using /data/mysql-binlog/mysql-binlog 080318 9:26:46 [Note] Starting crash recovery... 080318 9:26:46 [Note] Crash recovery finished. 080318 9:26:46 [Note] /usr/sbin/mysqld: ready for connections. Version: '5.0.27-standard-log' socket: '/var/lib/mysql/mysql.sock' port: 3306 MySQL Community Edition - Standard (GPL) 080318 09:27:06 mysqld started 080318 9:27:06 InnoDB: Database was not shut down normally! InnoDB: Starting crash recovery. InnoDB: Reading tablespace information from the .ibd files... InnoDB: Restoring possible half-written data pages from the doublewrite InnoDB: buffer... 080318 9:27:06 InnoDB: Starting log scan based on checkpoint at InnoDB: log sequence number 8 1783242805. InnoDB: Doing recovery: scanned up to log sequence number 8 1783269625 080318 9:27:06 InnoDB: Starting an apply batch of log records to the database... InnoDB: Progress in percents: 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 InnoDB: Apply batch completed InnoDB: Last MySQL binlog file position 0 27912, file name /data/mysql-binlog/mysql-binlog.000405 080318 9:27:07 InnoDB: Started; log sequence number 8 1783269625 080318 9:27:07 [Note] Recovering after a crash using /data/mysql-binlog/mysql-binlog 080318 9:27:07 [Note] Starting crash recovery... 080318 9:27:07 [Note] Crash recovery finished. 080318 9:27:07 [Note] /usr/sbin/mysqld: ready for connections. Version: '5.0.27-standard-log' socket: '/var/lib/mysql/mysql.sock' port: 3306 MySQL Community Edition - Standard (GPL) Status information: Current dir: /var/lib/mysql/ Running threads: 29 Stack size: 262144 Current locks: lock: 0x2c94358: lock: 0x2fe6648: lock: 0x2fe3eb8: lock: 0x2fe1a58: lock: 0x2fd9ed8: lock: 0x2f017b8: lock: 0x2ec51d0: lock: 0x2ebc630: lock: 0x2eb5b70: lock: 0x2eae980: lock: 0x2ea88a0: lock: 0x2ea2070: lock: 0x2e9d8f0: lock: 0x2e9ba80: lock: 0x2e97ec0: lock: 0x2e91e50: lock: 0x2e84900: lock: 0x2e73c70: lock: 0x2e6ae80: lock: 0x2bc8180: lock: 0x2e5f8d0: lock: 0x2e5ae60: lock: 0x2e57a20: lock: 0x2e4b060: lock: 0x2e49790: lock: 0x2e45d70: lock: 0x2e41210: lock: 0x2e3b5c0: lock: 0x2e28880: lock: 0x2e26810: lock: 0x2e21290: lock: 0x2e13340: lock: 0x2e0c490: lock: 0x2dff920: lock: 0x2df9840: lock: 0x2de19c0: lock: 0x2de7940: lock: 0x2decc20: lock: 0x2db3250: lock: 0x2dd2c80: lock: 0x2dc3c60: lock: 0x2db3750: lock: 0x2dbbde0: lock: 0x2da8e10: lock: 0x2d82cd0: lock: 0x2d8d580: lock: 0x2d5ae00: lock: 0x2d62b00: lock: 0x2d58000: lock: 0x2d57ef0: lock: 0x2d58190: lock: 0x2c51328: lock: 0x2c1e9f0: lock: 0x2c1ac40: lock: 0x2bfec08: lock: 0x2bf9138: lock: 0x2bf3e48: lock: 0x2be4c38: lock: 0x2be0c88: lock: 0x2bdac68: Key caches: default Buffer_size: 8388600 Block_size: 1024 Division_limit: 100 Age_limit: 300 blocks used: 13 not flushed: 0 w_requests: 0 writes: 0 r_requests: 49 reads: 13 handler status: read_key: 3465296 read_next: 3008659 read_rnd 73084 read_first: 8111 write: 17947 delete 1405 update: 0 Table status: Opened tables: 94 Open tables: 88 Open files: 28 Open streams: 0 Alarm status: Active alarms: 28 Max used alarms: 29 Next alarm time: 28155 080318 10:22:03 [Note] /usr/sbin/mysqld: Normal shutdown 080318 10:22:05 InnoDB: Starting shutdown... 080318 10:22:08 InnoDB: Shutdown completed; log sequence number 8 1787443746 080318 10:22:08 [Note] /usr/sbin/mysqld: Shutdown complete Memory status: Non-mmapped space allocated from system: 37298176 Number of free chunks: 64 Number of fastbin blocks: 0 Number of mmapped regions: 20 Space in mmapped regions: 1179205632 Maximum total allocated space: 0 Space available in freed fastbin blocks: 0 Total allocated space: 37105056 Total free space: 193120 Top-most, releasable space: 135072 Estimated memory (with thread stack): 1224105984 080318 10:22:08 mysqld ended 080318 10:22:09 mysqld started 080318 10:22:09 InnoDB: Started; log sequence number 8 1787443746 080318 10:22:09 [Note] /usr/sbin/mysqld: ready for connections. Version: '5.0.27-standard-log' socket: '/var/lib/mysql/mysql.sock' port: 3306 MySQL Community Edition - Standard (GPL)