Bug #24480 MYSQLD crashes after getting a error 4350 and 4006 when reading table
Submitted: 21 Nov 2006 17:05 Modified: 14 Jan 2007 20:58
Reporter: Kris Buytaert (Candidate Quality Contributor) Email Updates:
Status: No Feedback Impact on me:
None 
Category:MySQL Cluster: Cluster (NDB) storage engine Severity:S1 (Critical)
Version:5.1.11 OS:Linux (Centos 54.2)
Assigned to: CPU Architecture:Any

[21 Nov 2006 17:05] Kris Buytaert
Description:
mysql client stopped reacting after a show tables in a database.

Closer inspection showed the following 

061120 16:18:58  mysqld started
061120 16:18:59  InnoDB: Started; log sequence number 0 46403
061120 16:19:09 [Note] Starting MySQL Cluster Binlog Thread
061120 16:19:09 [Note] /usr/sbin/mysqld: ready for connections.
Version: '5.1.11-beta-log'  socket: '/tmp/mysql.sock'  port: 3306  MySQL Community Server (GPL)
061120 16:19:09 [Note] SCHEDULER: Manager thread booting
061120 16:19:09 [Note] SCHEDULER: Loaded 0 events
061120 16:19:09 [Note] SCHEDULER: Suspending operations
061121  6:18:32 [ERROR] Invalid (old?) table or database name 'lost+found'
061121  6:18:32 [ERROR] Invalid (old?) table or database name 'mysql-cluster'
061121  8:31:24 [ERROR] Got error 4350 when reading table './pmt/terminalderivedmetric'
061121  8:32:02 [ERROR] Got error 4350 when reading table './pmt/terminalderivedmetric'
061121  8:32:32 [ERROR] Got error 4006 when reading table './ccmt/isplistforipaddresspool'
mysqld got signal 11;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help diagnose
the problem, but since we have already crashed, something is definitely wrong
and this may fail.

key_buffer_size=8388600
read_buffer_size=1044480
max_used_connections=35
max_connections=100
threads_connected=32
It is possible that mysqld could use up to 
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_connections = 212591 K
bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

thd=0xdb9dfff0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
Cannot determine thread, fp=0x30661c, backtrace may not be correct.
Stack range sanity check OK, backtrace follows:
0x81d9210
0x8c88b8
(nil)
0x828c51e
0x828c322
0x81f015f
0x81f52b6
0x81f9abb
0x81f07e3
0x81f032d
0x81ef85f
0x8c3080
0x1f423e
New value of fp=(nil) failed sanity check, terminating stack trace!
Please read http://dev.mysql.com/doc/mysql/en/Using_stack_trace.html and follow instructions on how 
to resolve the stack trace. Resolved
stack trace is much more helpful in diagnosing the problem, so please do 
resolve it
Trying to get some variables.
Some pointers may be invalid and cause the dump to abort...
thd->query at 0x94620d8 = commit
thd->thread_id=9830
The manual page at http://www.mysql.com/doc/en/Crashing.html contains
information that should help you find out what is causing the crash.

Number of processes running now: 0

061121 08:33:00  mysqld restarted
061121  8:33:01  InnoDB: Started; log sequence number 0 46403
061121  8:33:01 [Note] Starting MySQL Cluster Binlog Thread
/usr/sbin/mysqld: Table 'general_log' is marked as crashed and should be repaired
/usr/sbin/mysqld: Table 'slow_log' is marked as crashed and should be repaired
061121  8:33:10 [Note] /usr/sbin/mysqld: ready for connections.
Version: '5.1.11-beta-log'  socket: '/tmp/mysql.sock'  port: 3306  MySQL Community Server (GPL)
061121  8:33:10 [Note] SCHEDULER: Manager thread booting
061121  8:33:10 [Note] SCHEDULER: Loaded 0 events
061121  8:33:10 [Note] SCHEDULER: Suspending operations
061121  8:34:49 [ERROR] Invalid (old?) table or database name 'lost+found'
061121  8:34:49 [ERROR] Invalid (old?) table or database name 'mysql-cluster'
061121  8:39:02 [ERROR] Invalid (old?) table or database name 'lost+found'
061121  8:39:02 [ERROR] Invalid (old?) table or database name 'mysql-cluster'
061121  9:25:19 [ERROR] Invalid (old?) table or database name 'lost+found'
061121  9:25:19 [ERROR] Invalid (old?) table or database name 'mysql-cluster'
061121  9:35:07 [Note] /usr/sbin/mysqld: Normal shutdown

061121  9:35:07 [Note] SCHEDULER: Killing manager thread 2
061121  9:35:07 [Note] SCHEDULER: Waiting the manager thread to reply
061121  9:35:07 [Note] SCHEDULER: Resuming operations
061121  9:35:07 [Note] SCHEDULER: Manager thread started with id 2
061121  9:35:07 [Note] SCHEDULER: Shutting down
061121  9:35:07 [Note] SCHEDULER: Waiting for worker threads to finish
061121  9:35:07 [Note] SCHEDULER: Emptying the queue
061121  9:35:07 [Note] SCHEDULER: Stopped
061121  9:35:09 [Warning] /usr/sbin/mysqld: Forcing close of thread 33  user: 'ccmt'

061121  9:35:09 [Warning] /usr/sbin/mysqld: Forcing close of thread 32  user: 'ccmt'

061121  9:35:09 [Warning] /usr/sbin/mysqld: Forcing close of thread 31  user: 'ccmt'

061121  9:35:09 [Warning] /usr/sbin/mysqld: Forcing close of thread 30  user: 'pmt'

Multiple warning , I eventually had to kill mysqld myselve.

Looks like Bug #18279 and #19176

Also the ndb nodes seem to stay up and running.  Connectinos from the other node keep working.

Trace : 

thd=0xdb9dfff0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
Cannot determine thread, fp=0x30661c, backtrace may not be correct.
Stack range sanity check OK, backtrace follows:
0x81d9210 handle_segfault + 416
0x8c88b8 (?)
(nil)
0x828c51e _Z19ha_commit_one_phaseP3THDb + 110
0x828c322 _Z15ha_commit_transP3THDb + 386
0x81f015f _Z9end_transP3THD25enum_mysql_completiontype + 95
0x81f52b6 _Z21mysql_execute_commandP3THD + 13382
0x81f9abb _Z11mysql_parseP3THDPcj + 347
0x81f07e3 _Z16dispatch_command19enum_server_commandP3THDPcj + 1123
0x81f032d _Z10do_commandP3THD + 141
0x81ef85f handle_one_connection + 655
0x8c3080 (?)
0x1f423e (?)
New value of fp=(nil) failed sanity check, terminating stack trace!
Please read http://dev.mysql.com/doc/mysql/en/Using_stack_trace.html and follow instructions on how 
to resolve the stack trace. Resolved
stack trace is much more helpful in diagnosing the problem, so please do 
resolve it
Trying to get some variables.

How to repeat:
not yet repeatable , only similar bugs
[22 Nov 2006 8:45] Valeriy Kravchuk
Thank you for a problem report. Please, try to repeat with a newer version, 5.1.12. In case of similar problems, please, send the results of SHOW CREATE TABLE and SHOW TABLE STATUS for those terminalderivedmetric and isplistforipaddresspool tables.
[22 Nov 2006 10:40] Kris Buytaert
I`m actually awaiting 5.1.13 binaries before I try the actual upgrade.
Will a rolling upgrade be feasable already ?
[14 Dec 2006 20:58] Valeriy Kravchuk
Please, try to repeat with a newer version, 5.1.14, already released. Inform about the results.
[15 Jan 2007 0:00] Bugs System
No feedback was provided for this bug for over a month, so it is
being suspended automatically. If you are able to provide the
information that was originally requested, please do so and change
the status of the bug back to "Open".