Description:
MySQL crashes frequently without any regularity.
It might occur when heavy flow during the day time and it also occur with low peak flow at night.
More than one slaves have crashed at different time.
We were using mysql 5.1.37, and the problem still exists after upgrade to 5.1.47.
How can I repeat or resolve this problem, please help me, thanks!
------------------------------
now database status:
/usr/local/webserver/mysql/bin/mysql Ver 14.14 Distrib 5.1.47, for unknown-linux-gnu (x86_64) using readline 5.1
Connection id: 276891261
Current database:
Current user: root@localhost
SSL: Not in use
Current pager: stdout
Using outfile: ''
Using delimiter: ;
Server version: 5.1.47-log Source distribution
Protocol version: 10
Connection: Localhost via UNIX socket
Server characterset: utf8
Db characterset: utf8
Client characterset: utf8
Conn. characterset: utf8
UNIX socket: /usr/local/webserver/mysql/tmp/mysql.sock
Uptime: 9 days 13 hours 42 min 6 sec
Threads: 2 Questions: 1415388186 Slow queries: 1843961 Opens: 23876884 Flush tables: 1 Open tables: 512 Queries per second avg: 1711.626
-----------------------------
MySQL error log when MySQL crashed
101023 3:40:01 - mysqld got signal 11 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help diagnose
the problem, but since we have already crashed, something is definitely wrong
and this may fail.
key_buffer_size=4294967296
read_buffer_size=2097152
max_used_connections=1477
max_threads=2000
threads_connected=9
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 12406897 K
bytes of memory
Hope that's ok; if not, decrease some variables in the equation.
thd: 0x2ab0009fb1b0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0x4d0720e0 thread_stack 0x40000
/usr/local/webserver/mysql/libexec/mysqld(my_print_stacktrace+0x24) [0x880d44]
/usr/local/webserver/mysql/libexec/mysqld(handle_segfault+0x322) [0x5d8782]
/lib64/libpthread.so.0 [0x316ee0eb10]
/usr/local/webserver/mysql/libexec/mysqld(ip_to_hostname(in_addr*, unsigned int*)+0x85) [0x5defd5]
/usr/local/webserver/mysql/libexec/mysqld(handle_one_connection+0x894) [0x5e0724]
/lib64/libpthread.so.0 [0x316ee0673d]
/lib64/libc.so.6(clone+0x6d) [0x316e2d3d1d]
Trying to get some variables.
Some pointers may be invalid and cause the dump to abort...
thd->query at (nil) is an invalid pointer
thd->thread_id=389533960
thd->killed=NOT_KILLED
The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you find out what is causing the crash.
101023 03:40:03 mysqld_safe Number of processes running now: 0
101023 03:40:03 mysqld_safe mysqld restarted
101023 3:40:03 [Warning] 'for replication startup options' is deprecated and will be removed in a future release. Please use ''CHAN
GE MASTER'' instead.
101023 3:40:03 [ERROR] Can't open shared library '/usr/local/webserver/mysql/lib/mysql/plugin/libmemcache_engine.so' (errno: 0 cann
ot open shared object file: No such file or directory)
101023 3:40:03 [Warning] Couldn't load plugin named 'memcache' with soname 'libmemcache_engine.so'.
InnoDB: The InnoDB memory heap is disabled
InnoDB: Mutexes and rw_locks use GCC atomic builtins
InnoDB: Compressed tables use zlib 1.2.3
101023 3:40:04 InnoDB: highest supported file format is Barracuda.
InnoDB: Log scan progressed past the checkpoint lsn 106568405073
101023 3:40:04 InnoDB: Database was not shut down normally!
InnoDB: Starting crash recovery.
InnoDB: Reading tablespace information from the .ibd files...
InnoDB: Restoring possible half-written data pages from the doublewrite
InnoDB: buffer...
InnoDB: Doing recovery: scanned up to log sequence number 106573647872
InnoDB: Doing recovery: scanned up to log sequence number 106578890752
InnoDB: Doing recovery: scanned up to log sequence number 106584133632
InnoDB: Doing recovery: scanned up to log sequence number 106589376512
InnoDB: Doing recovery: scanned up to log sequence number 106594619392
InnoDB: Doing recovery: scanned up to log sequence number 106599862272
InnoDB: Doing recovery: scanned up to log sequence number 106605105152
InnoDB: Doing recovery: scanned up to log sequence number 106610348032
InnoDB: Doing recovery: scanned up to log sequence number 106615590912
InnoDB: Doing recovery: scanned up to log sequence number 106620833792
InnoDB: Doing recovery: scanned up to log sequence number 106626076672
InnoDB: Doing recovery: scanned up to log sequence number 106631319552
InnoDB: Doing recovery: scanned up to log sequence number 106636562432
InnoDB: Doing recovery: scanned up to log sequence number 106641805312
InnoDB: Doing recovery: scanned up to log sequence number 106647048192
InnoDB: Doing recovery: scanned up to log sequence number 106652291072
InnoDB: Doing recovery: scanned up to log sequence number 106657533952
InnoDB: Doing recovery: scanned up to log sequence number 106662776832
InnoDB: Doing recovery: scanned up to log sequence number 106668019712
InnoDB: Doing recovery: scanned up to log sequence number 106673262592
InnoDB: Doing recovery: scanned up to log sequence number 106678505472
InnoDB: Doing recovery: scanned up to log sequence number 106683748352
InnoDB: Doing recovery: scanned up to log sequence number 106688991232
InnoDB: Doing recovery: scanned up to log sequence number 106694234112
InnoDB: Doing recovery: scanned up to log sequence number 106696444143
101023 3:40:17 InnoDB: Starting an apply batch of log records to the database...
InnoDB: Progress in percents: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 3
7 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 8
1 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99
InnoDB: Apply batch completed
InnoDB: Last MySQL binlog file position 0 515516, file name ./ms-mysql-bin.000003
101023 3:40:41 InnoDB Plugin 1.0.8 started; log sequence number 106696444143
101023 3:40:41 [Note] Recovering after a crash using ms-mysql-bin
101023 3:40:41 [Note] Starting crash recovery...
101023 3:40:41 [Note] Crash recovery finished.
101023 3:40:41 [Note] Slave SQL thread initialized, starting replication in log 'mysql-bin.001120' at position 529027904, relay log
'./ms-relay-bin.003373' position: 529028049
101023 3:40:41 [Note] Slave I/O thread: connected to master 'repl@192.168.113.201:3306',replication started in log 'mysql-bin.00112
0' at position 529029173
101023 3:40:41 [Note] Event Scheduler: Loaded 0 events
101023 3:40:41 [Note] /usr/local/webserver/mysql/libexec/mysqld: ready for connections.
Version: '5.1.47-log' socket: '/usr/local/webserver/mysql/tmp/mysql.sock' port: 3306 Source distribution
101023 3:40:47 [ERROR] /usr/local/webserver/mysql/libexec/mysqld: Table './mant/friend_link' is marked as crashed and should
be repaired
101023 3:40:47 [Warning] Checking table: './man/friend_link'
101023 3:40:48 [ERROR] /usr/local/webserver/mysql/libexec/mysqld: Table './man/help_question' is marked as crashed and shoul
d be repaired
How to repeat:
Can not repeat.
Suggested fix:
I don't know
