===================================== 2014-01-29 09:43:31 7f9b57fff700 INNODB MONITOR OUTPUT ===================================== Per second averages calculated from the last 20 seconds ----------------- BACKGROUND THREAD ----------------- srv_master_thread loops: 19425 srv_active, 0 srv_shutdown, 36093 srv_idle srv_master_thread log flush and writes: 55515 ---------- SEMAPHORES ---------- OS WAIT ARRAY INFO: reservation count 766929 --Thread 140305164658432 has waited at srv0srv.cc line 1998 for 952.00 seconds the semaphore: X-lock on RW-latch at 0x12538c0 created in file dict0dict.cc line 896 a writer (thread id 140322757281536) has reserved it in mode exclusive number of readers 0, waiters flag 1, lock_word: 0 Last time read locked in file row0purge.cc line 720 Last time write locked in file /var/tmp/portage/dev-db/mysql-5.6.15/work/mysql/storage/innobase/row/row0mysql.cc line 3745 --Thread 140305378477824 has waited at dict0dict.cc line 5391 for 675.00 seconds the semaphore: Mutex at 0x1a8d4bd8 created file dict0dict.cc line 887, lock var 1 waiters flag 1 --Thread 140305380075264 has waited at dict0dict.cc line 955 for 629.00 seconds the semaphore: Mutex at 0x1a8d4bd8 created file dict0dict.cc line 887, lock var 1 waiters flag 1 OS WAIT ARRAY INFO: signal count 14879749 Mutex spin waits 32841821, rounds 46231359, OS waits 345280 RW-shared spins 3755031, rounds 6000898, OS waits 47861 RW-excl spins 12378905, rounds 42755241, OS waits 300816 Spin rounds per wait: 1.41 mutex, 1.60 RW-shared, 3.45 RW-excl ------------ TRANSACTIONS ------------ Trx id counter 192564708 Purge done for trx's n:o < 192556796 undo n:o < 0 state: running but idle History list length 3086 LIST OF TRANSACTIONS FOR EACH SESSION: ---TRANSACTION 0, not started getting info on foreign keys MySQL thread id 12411, OS thread handle 0x7f9b643e8700, query id 170905 192.168.12.213 analyst Opening tables show create table `dbmain`.`network` ---TRANSACTION 192564645, not started MySQL thread id 12071, OS thread handle 0x7f9f701a0700, query id 170639 app151.domain 192.168.0.151 appuser checking permissions drop table if exists dbtemp.log_41_12071 ---TRANSACTION 192562820, not started mysql tables in use 1, locked 1 MySQL thread id 12069, OS thread handle 0x7f9b56f3a700, query id 170147 app77.domain 192.168.0.77 appuser closing tables insert ignore into dblog.log_28 select * from dbtemp.log_41_12069 ---TRANSACTION 192564703, ACTIVE 954 sec dropping table 18 lock struct(s), heap size 3112, 51 row lock(s), undo log entries 36 MySQL thread id 12071, OS thread handle 0x7f9f701a0700, query id 170639 app151.domain 192.168.0.151 appuser checking permissions drop table if exists dbtemp.log_41_12071 Trx read view will not see trx with id >= 192564708, sees < 192564708 -------- FILE I/O -------- I/O thread 0 state: waiting for i/o request (insert buffer thread) I/O thread 1 state: waiting for i/o request (log thread) I/O thread 2 state: waiting for i/o request (read thread) I/O thread 3 state: waiting for i/o request (read thread) I/O thread 4 state: waiting for i/o request (read thread) I/O thread 5 state: waiting for i/o request (read thread) I/O thread 6 state: waiting for i/o request (write thread) I/O thread 7 state: waiting for i/o request (write thread) I/O thread 8 state: waiting for i/o request (write thread) I/O thread 9 state: waiting for i/o request (write thread) Pending normal aio reads: 0 [0, 0, 0, 0] , aio writes: 0 [0, 0, 0, 0] , ibuf aio reads: 0, log i/o's: 0, sync i/o's: 0 Pending flushes (fsync) log: 0; buffer pool: 0 6750805 OS file reads, 7020786 OS file writes, 1173401 OS fsyncs 4 pending preads, 0 pending pwrites 355.28 reads/s, 16384 avg bytes/read, 0.00 writes/s, 0.00 fsyncs/s ------------------------------------- INSERT BUFFER AND ADAPTIVE HASH INDEX ------------------------------------- Ibuf: size 1, free list len 393223, seg size 393225, 2899525 merges merged operations: insert 2470144, delete mark 1061220, delete 13966 discarded operations: insert 0, delete mark 0, delete 0 Hash table size 29749513, node heap has 5024 buffer(s) 959.05 hash searches/s, 842.21 non-hash searches/s --- LOG --- Log sequence number 4893228750761 Log flushed up to 4893228750761 Pages flushed up to 4893228750761 Last checkpoint at 4893228299430 0 pending log writes, 0 pending chkp writes 276338 log i/o's done, 0.00 log i/o's/second ---------------------- BUFFER POOL AND MEMORY ---------------------- Total memory allocated 15384707072; in additional pool allocated 0 Dictionary memory allocated 7026386 Buffer pool size 917496 Free buffers 0 Database pages 912472 Old database pages 336672 Modified db pages 0 Pending reads 4 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 1861432, not young 8519987 13.15 youngs/s, 681.32 non-youngs/s Pages read 7008463, created 1577786, written 9603993 355.28 reads/s, 0.00 creates/s, 0.00 writes/s Buffer pool hit rate 924 / 1000, young-making rate 2 / 1000 not 147 / 1000 Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 912472, unzip_LRU len: 0 I/O sum[1934248]:cur[78728], unzip sum[0]:cur[0] ---------------------- INDIVIDUAL BUFFER POOL INFO ---------------------- ---BUFFER POOL 0 Buffer pool size 114687 Free buffers 0 Database pages 114053 Old database pages 42082 Modified db pages 0 Pending reads 1 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 236681, not young 1138772 1.70 youngs/s, 77.15 non-youngs/s Pages read 881505, created 202424, written 1337338 42.65 reads/s, 0.00 creates/s, 0.00 writes/s Buffer pool hit rate 929 / 1000, young-making rate 2 / 1000 not 129 / 1000 Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 114053, unzip_LRU len: 0 I/O sum[241781]:cur[9841], unzip sum[0]:cur[0] ---BUFFER POOL 1 Buffer pool size 114687 Free buffers 0 Database pages 114093 Old database pages 42096 Modified db pages 0 Pending reads 1 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 228354, not young 1047755 1.20 youngs/s, 82.85 non-youngs/s Pages read 864125, created 196932, written 1165016 45.90 reads/s, 0.00 creates/s, 0.00 writes/s Buffer pool hit rate 917 / 1000, young-making rate 2 / 1000 not 150 / 1000 Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 114093, unzip_LRU len: 0 I/O sum[241781]:cur[9841], unzip sum[0]:cur[0] ---BUFFER POOL 2 Buffer pool size 114687 Free buffers 0 Database pages 114047 Old database pages 42080 Modified db pages 0 Pending reads 1 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 233658, not young 1073403 1.30 youngs/s, 71.90 non-youngs/s Pages read 874896, created 194992, written 1177887 41.10 reads/s, 0.00 creates/s, 0.00 writes/s Buffer pool hit rate 857 / 1000, young-making rate 4 / 1000 not 250 / 1000 Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 114047, unzip_LRU len: 0 I/O sum[241781]:cur[9841], unzip sum[0]:cur[0] ---BUFFER POOL 3 Buffer pool size 114687 Free buffers 0 Database pages 114051 Old database pages 42081 Modified db pages 0 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 233198, not young 1048325 1.65 youngs/s, 85.05 non-youngs/s Pages read 876122, created 195413, written 1185835 47.15 reads/s, 0.00 creates/s, 0.00 writes/s Buffer pool hit rate 948 / 1000, young-making rate 1 / 1000 not 94 / 1000 Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 114051, unzip_LRU len: 0 I/O sum[241781]:cur[9841], unzip sum[0]:cur[0] ---BUFFER POOL 4 Buffer pool size 114687 Free buffers 0 Database pages 114054 Old database pages 42082 Modified db pages 0 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 234172, not young 1053031 1.60 youngs/s, 83.45 non-youngs/s Pages read 879313, created 196602, written 1205933 45.15 reads/s, 0.00 creates/s, 0.00 writes/s Buffer pool hit rate 889 / 1000, young-making rate 3 / 1000 not 206 / 1000 Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 114054, unzip_LRU len: 0 I/O sum[241781]:cur[9841], unzip sum[0]:cur[0] ---BUFFER POOL 5 Buffer pool size 114687 Free buffers 0 Database pages 114048 Old database pages 42080 Modified db pages 0 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 230732, not young 1059286 2.00 youngs/s, 79.55 non-youngs/s Pages read 881204, created 197691, written 1166622 44.30 reads/s, 0.00 creates/s, 0.00 writes/s Buffer pool hit rate 854 / 1000, young-making rate 6 / 1000 not 263 / 1000 Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 114048, unzip_LRU len: 0 I/O sum[241781]:cur[9841], unzip sum[0]:cur[0] ---BUFFER POOL 6 Buffer pool size 114687 Free buffers 0 Database pages 114061 Old database pages 42085 Modified db pages 0 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 233480, not young 1041795 1.60 youngs/s, 88.00 non-youngs/s Pages read 881129, created 196949, written 1160087 45.90 reads/s, 0.00 creates/s, 0.00 writes/s Buffer pool hit rate 957 / 1000, young-making rate 1 / 1000 not 82 / 1000 Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 114061, unzip_LRU len: 0 I/O sum[241781]:cur[9841], unzip sum[0]:cur[0] ---BUFFER POOL 7 Buffer pool size 114687 Free buffers 0 Database pages 114065 Old database pages 42086 Modified db pages 0 Pending reads 1 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 231157, not young 1057620 2.10 youngs/s, 113.39 non-youngs/s Pages read 870169, created 196783, written 1205275 43.15 reads/s, 0.00 creates/s, 0.00 writes/s Buffer pool hit rate 917 / 1000, young-making rate 4 / 1000 not 219 / 1000 Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 114065, unzip_LRU len: 0 I/O sum[241781]:cur[9841], unzip sum[0]:cur[0] -------------- ROW OPERATIONS -------------- 0 queries inside InnoDB, 0 queries in queue 5 read views open inside InnoDB Main thread process no. 1793, id 140305164658432, state: enforcing dict cache limit Number of rows inserted 85387711, updated 0, deleted 451394, read 19840599 0.00 inserts/s, 0.00 updates/s, 0.00 deletes/s, 3845.26 reads/s ---------------------------- END OF INNODB MONITOR OUTPUT ============================ InnoDB: ###### Diagnostic info printed to the standard error stream InnoDB: Error: semaphore wait has lasted > 600 seconds InnoDB: We intentionally crash the server, because it appears to be hung. 2014-01-29 09:43:33 7f9b64df1700 InnoDB: Assertion failure in thread 140305389000448 in file srv0srv.cc line 1748 InnoDB: We intentionally generate a memory trap. InnoDB: Submit a detailed bug report to http://bugs.mysql.com. InnoDB: If you get repeated assertion failures or crashes, even InnoDB: immediately after the mysqld startup, there may be InnoDB: corruption in the InnoDB tablespace. Please refer to InnoDB: http://dev.mysql.com/doc/refman/5.6/en/forcing-innodb-recovery.html InnoDB: about forcing recovery. 14:43:33 UTC - mysqld got signal 6 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. key_buffer_size=1073741824 read_buffer_size=2097152 max_used_connections=38 max_threads=250 thread_count=20 connection_count=20 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 34331992 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x0 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0 thread_stack 0x40000 /usr/sbin/mysqld(my_print_stacktrace+0x27)[0x875e74] /usr/sbin/mysqld(handle_fatal_signal+0x368)[0x636bdc] /lib64/libpthread.so.0(+0x10bf0)[0x7f9f74b41bf0] /lib64/libc.so.6(gsignal+0x35)[0x7f9f741c6c15] /lib64/libc.so.6(abort+0x17b)[0x7f9f741c808b] /usr/sbin/mysqld[0x9b8d9a] /lib64/libpthread.so.0(+0x8ec6)[0x7f9f74b39ec6] /lib64/libc.so.6(clone+0x6d)[0x7f9f7427998d] The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash.