Bug #17381 HA Testing: Node failure network port closed sometimes reports ndbrequire fail
Submitted: 14 Feb 2006 15:23 Modified: 8 Aug 2006 18:38
Reporter: Jonathan Miller Email Updates:
Status: Closed Impact on me:
None 
Category:MySQL Cluster: Cluster (NDB) storage engine Severity:S3 (Non-critical)
Version:5.1.7 OS:Linux (Linux 32 Bit OS)
Assigned to: Jonas Oreland CPU Architecture:Any

[14 Feb 2006 15:23] Jonathan Miller
Description:
In testing patch for http://bugs.mysql.com/bug.php?id=17325, I found that some of the time the node shutdown due to missed heartbeat would report:

Time: Monday 13 February 2006 - 17:51:02
Status: Temporary error, restart node
Message: Internal program error (failed ndbrequire) (Internal error, programming error or missing error message, please report a bug)
Error: 2341
Error data:
Error object: QMGR (Line: 2118) 0x0000000e
Program: /home/ndbdev/jmiller/builds/libexec/ndbd
Pid: 29119
Trace: /space/run/ndb_2_trace.log.5
Version: Version 5.1.7 (beta)
***EOM***

Other times it would report:

Time: Monday 13 February 2006 - 17:52:48
Status: Temporary error, restart node
Message: Arbitrator shutdown, please investigate error(s) on other node(s) (Arbitration error)
Error: 2305
Error data: Arbitrator decided to shutdown this node
Error object: QMGR (Line: 3826) 0x0000000a
Program: /home/ndbdev/jmiller/builds/libexec/ndbd
Pid: 29188
Trace: /space/run/ndb_2_trace.log.6
Version: Version 5.1.7 (beta)
***EOM***
This was repeated several times on a different node

Time: Monday 13 February 2006 - 17:54:01
Status: Temporary error, restart node
Message: Internal program error (failed ndbrequire) (Internal error, programming error or missing error message, please report a bug)
Error: 2341
Error data:
Error object: QMGR (Line: 2118) 0x0000000e
Program: /home/ndbdev/jmiller/builds/libexec/ndbd
Pid: 12627
Trace: /space/run/ndb_3_trace.log.3
Version: Version 5.1.7 (beta)
***EOM***

Time: Monday 13 February 2006 - 17:54:51
Status: Temporary error, restart node
Message: Internal program error (failed ndbrequire) (Internal error, programming error or missing error message, please report a bug)
Error: 2341
Error data: dblqh/DblqhMain.cpp
Error object: DBLQH (Line: 8592) 0x0000000a
Program: /home/ndbdev/jmiller/builds/libexec/ndbd
Pid: 12736
Trace: /space/run/ndb_3_trace.log.4
Version: Version 5.1.7 (beta)
***EOM***
Time: Monday 13 February 2006 - 17:56:44
Status: Temporary error, restart node
Message: Arbitrator shutdown, please investigate error(s) on other node(s) (Arbitration error)
Error: 2305
Error data: Arbitrator decided to shutdown this node
Error object: QMGR (Line: 3826) 0x0000000a
Program: /home/ndbdev/jmiller/builds/libexec/ndbd
Pid: 12785
Trace: /space/run/ndb_3_trace.log.5
Version: Version 5.1.7 (beta)
***EOM***

It should be noted that the original bug #17325 was no longer reproduced and that the issues with the data were not found once the patch was applied. Per the lead developer all ndbrequire error messages need to be reported.

How to repeat:
Configuration and steps recorded in http://bugs.mysql.com/bug.php?id=17325
[18 Apr 2006 6:03] Jonas Oreland
The LQH is occuring from occationally in autotest.
Closing this bug as it's about something else.
[18 Apr 2006 10:19] Jonas Oreland
qmgr stuff is fixed.
scan/lqh is a different bug, reopen if you get it again