Bug #73339 | NDB data node crashes after upgrade from 7.2.13 to 7.3.5 - illegal signal | ||
---|---|---|---|
Submitted: | 21 Jul 2014 9:54 | Modified: | 23 Mar 2016 14:05 |
Reporter: | Christian Ehmig | Email Updates: | |
Status: | No Feedback | Impact on me: | |
Category: | MySQL Cluster: Cluster (NDB) storage engine | Severity: | S1 (Critical) |
Version: | 7.3.6 | OS: | Linux (2.6.32) |
Assigned to: | MySQL Verification Team | CPU Architecture: | Any |
[21 Jul 2014 9:54]
Christian Ehmig
[24 Jul 2014 9:49]
Christian Ehmig
Upgraded the cluster (data and mgm nodes only) to 7.3.6. Nodes keep crashing. Time: Thursday 24 July 2014 - 10:31:41 Status: Temporary error, restart node Message: WatchDog terminate, internal error or massive overload on the machine running this node (Internal error, programming error or missing error message, please report a bug) Error: 6050 Error data: Job Handling Error object: /export/home/pb2/build/sb_0-12598553-1404311822.99/mysql-cluster-gpl-7.3.6/storage/ndb/src/kernel/vm/WatchDog.cpp Program: ndbmtd Pid: 7188 Version: mysql-5.6.19 ndb-7.3.6
[24 Jul 2014 9:50]
Christian Ehmig
New errors on other nodes: (Version 7.3.5) tatus: Temporary error, restart node Message: Assertion (Internal error, programming error or missing error message, please report a bug) Error: 2301 Error data: Illegal signal received (GSN 40 not added) Error object: Illegal signal received (GSN 40 not added) Program: ndbmtd Pid: 6029 thr: 0 Version: mysql-5.6.17 ndb-7.3.5 Trace: /mnt/data/cluster/ndb_6_trace.log.8 [t1..t10] (Version 7.3.6) Time: Thursday 24 July 2014 - 08:01:49 Status: Temporary error, restart node Message: Internal program error (failed ndbrequire) (Internal error, programming error or missing error message, please report a bug) Error: 2341 Error data: DblqhMain.cpp Error object: DBLQH (Line: 8862) 0x00000006 Program: ndbmtd Pid: 19753 thr: 3 Version: mysql-5.6.19 ndb-7.3.6 Trace: /mnt/data/cluster/ndb_6_trace.log.9 [t1..t10] ***EOM***
[24 Jul 2014 10:00]
Christian Ehmig
ndb config
Attachment: ndb_mgmd.cnf (application/octet-stream, text), 3.87 KiB.
[24 Jul 2014 10:00]
Christian Ehmig
node 6 trace 8
Attachment: ndb_6_trace.log.8.tgz (application/x-compressed, text), 617.83 KiB.
[24 Jul 2014 10:00]
Christian Ehmig
node 6 trace 9
Attachment: ndb_6_trace.log.9.tgz (application/x-compressed, text), 930.47 KiB.
[24 Jul 2014 10:17]
Christian Ehmig
ndb 3 trace 9
Attachment: ndb_3_trace.log.9.tgz (application/x-compressed, text), 575.94 KiB.
[24 Jul 2014 22:11]
Christian Ehmig
Whole cluster crashed again (better to say, the two remaining nodes crashed after the first two nodes crashed this morning). Time: Thursday 24 July 2014 - 22:13:35 Status: Temporary error, restart node Message: Internal program error (failed ndbrequire) (Internal error, programming error or missing error message, please report a bug) Error: 2341 Error data: DblqhMain.cpp Error object: DBLQH (Line: 8862) 0x00000006 Program: ndbmtd Pid: 22137 thr: 3 Version: mysql-5.6.19 ndb-7.3.6 Trace: /mnt/data/cluster/ndb_4_trace.log.9 [t1..t10] ***EOM***
[23 Feb 2016 14:05]
MySQL Verification Team
Hi, the (Version 7.3.5) tatus: Temporary error, restart node Message: Assertion (Internal error, programming error or missing error message, please report a bug) Error: 2301 Error data: Illegal signal received (GSN 40 not added) Error object: Illegal signal received (GSN 40 not added) Program: ndbmtd Pid: 6029 thr: 0 Version: mysql-5.6.17 ndb-7.3.5 Trace: /mnt/data/cluster/ndb_6_trace.log.8 [t1..t10] crash is due to a bug fixed in 7.3.6 (18455971). The other crashes I don't have enough information to see why they happen. Do you have a way to reproduce them? This BUG report is rather old, did you see these crashes happen again since? On your latest installation? all best Bogdan Kecman
[23 Feb 2016 14:07]
MySQL Verification Team
The Time: Thursday 24 July 2014 - 10:31:41 Status: Temporary error, restart node Message: WatchDog terminate, internal error or massive overload on the machine running this node (Internal error, programming error or missing error message, please report a bug) Error: 6050 Error data: Job Handling Error object: /export/home/pb2/build/sb_0-12598553-1404311822.99/mysql-cluster-gpl-7.3.6/storage/ndb/src/kernel/vm/WatchDog.cpp Program: ndbmtd Pid: 7188 Version: mysql-5.6.19 ndb-7.3.6 crash is due to overload of the system or miss configuration of one. To fix this one I suggest you contact Oracle MySQL Cluster Support team. all best Bogdan Kecman
[24 Mar 2016 1:00]
Bugs System
No feedback was provided for this bug for over a month, so it is being suspended automatically. If you are able to provide the information that was originally requested, please do so and change the status of the bug back to "Open".