Description:
one of ndbd nodes is forced to shutdown when doing
alter online table X reorganize partition
(table X is the disk table)
Note: the problem happens only when I run the test case(attachment) I wrote using mtr, but not happened when doing manual test.
In the cluster log:
2009-03-05 19:03:40 [MgmSrvr] INFO -- Node 1: reorg-copy table 14 processed
49950 rows
2009-03-05 19:03:44 [MgmSrvr] INFO -- Node 2: reorg-copy table 14 processed
50050 rows
2009-03-05 19:03:55 [MgmSrvr] INFO -- Node 40: reorg-delete table 14 processed 0 rows
2009-03-05 19:03:55 [MgmSrvr] INFO -- Node 41: reorg-delete table 14 processed 0 rows
2009-03-05 19:04:24 [MgmSrvr] ALERT -- Node 2: Node 1 Disconnected
2009-03-05 19:04:24 [MgmSrvr] INFO -- Node 2: Communication to Node 1 closed2009-03-05 19:04:24 [MgmSrvr] ALERT -- Node 3: Node 1 Disconnected
2009-03-05 19:04:24 [MgmSrvr] ALERT -- Node 2: Arbitration check won - node group majority
2009-03-05 19:04:24 [MgmSrvr] INFO -- Node 2: President restarts arbitration thread [state=6]
2009-03-05 19:04:24 [MgmSrvr] INFO -- Node 2: GCP Take over started
2009-03-05 19:04:24 [MgmSrvr] INFO -- Node 2: Node 2 taking over as DICT master
2009-03-05 19:04:24 [MgmSrvr] ALERT -- Node 1: Forced node shutdown completed. Caused by error 2341: 'Internal program error (failed ndbrequire)(Internal error, programming error or missing error message, please report a bug). Temporary
error, restart node'.
The error log says:
Message: Internal program error (failed ndbrequire) (Internal error, programming error or missing error message, please report a bug)
Error: 2341
Error data: dbdict/Dbdict.cpp
Error object: DBDICT (Line: 24138) 0x00000002
Program: /home/wx228566/mysql/new/mysql-5.1-telco-6.4/run/libexec/ndbd
Pid: 25308
Trace: /home/wx228566/mysql/new/mysql-5.1-telco-6.4/run/mysql-test/var/mysql_cluster.1/ndbd.1/ndb_1_trace.log.1
How to repeat:
/mtr.pl --do-test=ndb_add_node_DD03
1 put the next four files in diectory mysql-test/include
ndb_add_node_mysqld.inc
ndb_add_node.inc
reload_ndb_mgmd.inc
add_two_nodes.inc
2 put restart_mysqld.inc in directory mysql-test/include instead of the existed one
3 put my.cnf in suite/ndb/my.cnf instead of the existed one
4 put table_data100000.dat in directory mysql-test/std_data
5 put test case ndb_add_node_DD03.test in suite/ndb/t/
what the test case does:
create disk table and load data to tables
add two nodes and restart cluster with four nodes running
alter online table X reorganize partition