Description:
Hi,
Cluster running with no activity :
2012-08-21 18:35:22 [ndbd] INFO -- findNeighbours from: 4861 old (left: 3 right: 3) new (65535 65535)
REMOVING lcp: 98 from table: 2 frag: 0 node: 3
REMOVING lcp: 98 from table: 2 frag: 1 node: 3
REMOVING lcp: 98 from table: 2 frag: 2 node: 3
REMOVING lcp: 98 from table: 2 frag: 3 node: 3
REMOVING lcp: 98 from table: 2 frag: 4 node: 3
REMOVING lcp: 98 from table: 2 frag: 5 node: 3
REMOVING lcp: 98 from table: 2 frag: 6 node: 3
start_resend(0, empty bucket (92682/4 92682/3) -> activeREMOVING lcp: 98 from table: 2 frag: 7 node: 3
Finished with handling node-failure
execGCP_NOMORETRANS(92682/4) c_ongoing_take_over_cnt -> seize
2012-08-21 18:35:22 [ndbd] INFO -- Illegal signal received (GSN 36 not added)
2012-08-21 18:35:22 [ndbd] INFO -- Illegal signal received (GSN 36 not added)
2012-08-21 18:35:22 [ndbd] INFO -- Error handler shutting down system
2012-08-21 18:35:22 [ndbd] INFO -- Error handler shutdown completed - exiting
2012-08-21 18:35:23 [ndbd] ALERT -- Node 4: Forced node shutdown completed. Caused by error 2301: 'Assertion(Internal error, programming error or missing error message, please report a bug). Temporary error, restart node'.
Error message:
Time: Tuesday 21 August 2012 - 18:35:21
Status: Temporary error, restart node
Message: Assertion (Internal error, programming error or missing error message, please report a bug)
Error: 2301
Error data: Illegal signal received (GSN 36 not added)
Error object: Illegal signal received (GSN 36 not added)
Program: ndbmtd
Pid: 1710 thr: 0
Version: mysql-5.5.22 ndb-7.2.6
Trace: /usr/local/mysql/data/ndb_3_trace.log.2 [t1..t8]
***EOM***
Trace :
NDBFS 001546 001379
NDBFS 001518 001520 001379 001537
QMGR 000121 000145 002626 002668 002677
--------------- Signal ----------------
r.bn: 245 "DBTC", r.proc: 3, r.sigId: 16531317 gsn: 36 "TCRELEASEREQ" prio: 1
s.bn: 32774 "API", s.proc: 21, s.sigId: 0 length: 3 trace: 1 #sec: 0 fragInf: 0
H'00000093 H'80060015 H'00000004
Source : ./storage/ndb/src/kernel/blocks/qmgr/QmgrMain.cpp
2667 if (cactivateApiCheck != 0) {
2668 jam();
2669 if (clatestTransactionCheck == 0) {
2670 //-------------------------------------------------------------
2671 // Initialise the Transaction check timer.
2672 //-------------------------------------------------------------
2673 clatestTransactionCheck = TcurrentTime;
2674 }//if
2675 int counter = 0;
2676 while (TcurrentTime > ((NDB_TICKS)10 + clatestTransactionCheck)) {
2677 jam();
2678 clatestTransactionCheck += (NDB_TICKS)10;
2679 sendSignal(DBTC_REF, GSN_TIME_SIGNAL, signal, 1, JBB);
2680 sendSignal(DBLQH_REF, GSN_TIME_SIGNAL, signal, 1, JBB);
2681 counter++;
2682 if (counter > 1) {
2683 jam();
2684 break;
2685 } else {
2686 ;
2687 }//if
2688 }//while
2689 }//if
2690
I have no clue on how to reproduce it, or investigate it further
I am not using 7.2.7 because of online backup problem.
How to repeat:
I have no clue on how to reproduce it, this happened 20 minutes after we stopped a sysbench mixed-oltp benchmark.