Description:
Mysql unexpectedly falls during hight load operations. MySQL runs in jail on ZFS.
from log
111017 01:28:34 mysqld_safe mysqld restarted
111017 1:28:34 InnoDB: The InnoDB memory heap is disabled
111017 1:28:34 InnoDB: Mutexes and rw_locks use GCC atomic builtins
111017 1:28:34 InnoDB: Compressed tables use zlib 1.2.3
111017 1:28:34 InnoDB: Initializing buffer pool, size = 8.0G
111017 1:28:35 InnoDB: Completed initialization of buffer pool
111017 1:28:35 InnoDB: highest supported file format is Barracuda.
InnoDB: The log sequence number in ibdata files does not match
InnoDB: the log sequence number in the ib_logfiles!
111017 1:28:35 InnoDB: Database was not shut down normally!
InnoDB: Starting crash recovery.
InnoDB: Reading tablespace information from the .ibd files...
InnoDB: Restoring possible half-written data pages from the doublewrite
InnoDB: buffer...
InnoDB: Last MySQL binlog file position 0 701, file name ./mysql-bin.000108
111017 1:28:35 InnoDB: Waiting for the background threads to start
111017 1:28:36 InnoDB: 1.1.8 started; log sequence number 19649871432
111017 1:28:36 [Note] Recovering after a crash using mysql-bin
111017 1:28:36 [Note] Starting crash recovery...
111017 1:28:36 [Note] Crash recovery finished.
111017 1:28:36 [Note] Event Scheduler: Loaded 4 events
111017 1:28:36 [Note] /usr/local/libexec/mysqld: ready for connections.
Version: '5.5.16-log' socket: '/tmp/mysql.sock' port: 3306 Source distribution
111017 1:28:36 [Note] Event Scheduler: scheduler thread started with id 1
111017 1:28:37 [Note] Start binlog_dump to slave_server(2), pos(mysql-bin.000108, 701)
len 232; hex 784b8e310a00000097c0ec2a08000000800c510708000000280f72310a00000000000000000000000000000000000000030000000a00000002000000000000000100000000000000020000000000000001000000000000000200000000000000000000000000000000000000000000000000000000000000ffffffffffffffff000000000000000001000000000000008344510700000000fdd970310a00000002000000000000000100000000000000800c510708000000000000000000000060e111770000000002000000000000000000000000000000f8d970310a0000001000000000000000; asc xK 1 * Q ( r1 DQ p1 Q ` w p1 ;
111017 1:29:30 InnoDB: Assertion failure in thread 43574098944 in file btr0pcur.c line 236
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: http://dev.mysql.com/doc/refman/5.5/en/forcing-innodb-recovery.html
InnoDB: about forcing recovery.
111017 1:29:30 - mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help diagnose
the problem, but since we have already crashed, something is definitely wrong
and this may fail.
key_buffer_size=8388608
read_buffer_size=8388608
max_used_connections=8
max_threads=400
thread_count=5
connection_count=4
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 9843023 K
bytes of memory
Hope that's ok; if not, decrease some variables in the equation.
cat /var/db/mysql/sql0.1september.ru.err | grep "file btr0pcur.c line 236"
111016 1:22:13 InnoDB: Assertion failure in thread 43036229440 in file btr0pcur.c line 236
111016 1:43:28 InnoDB: Assertion failure in thread 43036220416 in file btr0pcur.c line 236
111016 2:14:23 InnoDB: Assertion failure in thread 52941928896 in file btr0pcur.c line 236
111016 2:38:40 InnoDB: Assertion failure in thread 52941924864 in file btr0pcur.c line 236
111016 2:39:37 InnoDB: Assertion failure in thread 52941868608 in file btr0pcur.c line 236
111016 2:43:28 InnoDB: Assertion failure in thread 52942087040 in file btr0pcur.c line 236
111016 2:56:29 InnoDB: Assertion failure in thread 53319052928 in file btr0pcur.c line 236
111016 3:00:26 InnoDB: Assertion failure in thread 52942366144 in file btr0pcur.c line 236
111016 3:05:42 InnoDB: Assertion failure in thread 52942376576 in file btr0pcur.c line 236
111016 3:07:30 InnoDB: Assertion failure in thread 52942003456 in file btr0pcur.c line 236
111016 3:13:37 InnoDB: Assertion failure in thread 52942162752 in file btr0pcur.c line 236
111016 3:28:52 InnoDB: Assertion failure in thread 52941962496 in file btr0pcur.c line 236
111016 3:44:18 InnoDB: Assertion failure in thread 52707078784 in file btr0pcur.c line 236
111016 15:35:50 InnoDB: Assertion failure in thread 53499943744 in file btr0pcur.c line 236
111016 15:51:29 InnoDB: Assertion failure in thread 52707238720 in file btr0pcur.c line 236
111016 15:51:38 InnoDB: Assertion failure in thread 52707226048 in file btr0pcur.c line 236
111016 15:52:39 InnoDB: Assertion failure in thread 52707231872 in file btr0pcur.c line 236
111016 15:53:18 InnoDB: Assertion failure in thread 52707228288 in file btr0pcur.c line 236
111016 15:54:50 InnoDB: Assertion failure in thread 53085180864 in file btr0pcur.c line 236
111016 15:55:30 InnoDB: Assertion failure in thread 52707231872 in file btr0pcur.c line 236
111016 16:00:36 InnoDB: Assertion failure in thread 52740607552 in file btr0pcur.c line 236
111016 16:00:51 InnoDB: Assertion failure in thread 53118621568 in file btr0pcur.c line 236
111016 16:00:57 InnoDB: Assertion failure in thread 52740794944 in file btr0pcur.c line 236
111016 16:11:00 InnoDB: Assertion failure in thread 52740784512 in file btr0pcur.c line 236
111016 16:11:56 InnoDB: Assertion failure in thread 52740796288 in file btr0pcur.c line 236
111016 16:17:30 InnoDB: Assertion failure in thread 48492709248 in file btr0pcur.c line 236
111016 16:22:49 InnoDB: Assertion failure in thread 48189876288 in file btr0pcur.c line 236
111016 16:23:01 InnoDB: Assertion failure in thread 48190035648 in file btr0pcur.c line 236
111016 16:25:46 InnoDB: Assertion failure in thread 48190032512 in file btr0pcur.c line 236
111016 16:25:53 InnoDB: Assertion failure in thread 48189880832 in file btr0pcur.c line 236
111016 16:27:48 InnoDB: Assertion failure in thread 48156318720 in file btr0pcur.c line 236
111016 16:30:08 InnoDB: Assertion failure in thread 48156481664 in file btr0pcur.c line 236
111016 16:47:45 InnoDB: Assertion failure in thread 48589033664 in file btr0pcur.c line 236
111016 16:48:03 InnoDB: Assertion failure in thread 48503089280 in file btr0pcur.c line 236
111016 16:59:31 InnoDB: Assertion failure in thread 48571865152 in file btr0pcur.c line 236
111016 16:59:35 InnoDB: Assertion failure in thread 48156480320 in file btr0pcur.c line 236
111016 16:59:40 InnoDB: Assertion failure in thread 48156321472 in file btr0pcur.c line 236
111017 1:17:34 InnoDB: Assertion failure in thread 43574096640 in file btr0pcur.c line 236
111017 1:18:05 InnoDB: Assertion failure in thread 43574102528 in file btr0pcur.c line 236
111017 1:28:14 InnoDB: Assertion failure in thread 43574243712 in file btr0pcur.c line 236
111017 1:28:34 InnoDB: Assertion failure in thread 43574248192 in file btr0pcur.c line 236
111017 1:29:30 InnoDB: Assertion failure in thread 43574098944 in file btr0pcur.c line 236
I have a lot of INNODB tables, several of them are big.
How to repeat:
I make mysqldump of all databases when MySQL server is on load.
