Bug #47530 Node restart fails with ndbmtd and >2 replicas

• Submitted: 23 Sep 2009 0:01
• Modified: 23 Sep 2009 9:56
• Reporter: David Enbal
• Status: Closed
• Category: MySQL Cluster: Cluster (NDB) storage engine
• Severity: S1 (Critical)
• Version: 7.0.7
• OS: Any
• Assigned to: Jonas Oreland
• CPU Architecture: Any

[23 Sep 2009 0:01] David Enbal

Description:
I have a 3 data node cluster. Rebooting one of the nodes gets the entire cluster to crash.

The error message is:
Time: Wednesday 23 September 2009 - 02:52:21
Status: Temporary error, restart node
Message: Assertion (Internal error, programming error or missing error message, please report a bug)
Error: 2301
Error data: Illegal signal received (GSN 70 not added)
Error object: Illegal signal received (GSN 70 not added)
Program: /opt/mysql/mysql/bin/ndbmtd
Pid: 3842
Trace: /opt/mysql/mysql/data/ndb_4_trace.log.10 /opt/mysql/mysql/data/ndb_4_trace.log.10_t1 /opt/mysql/mysql/data/ndb_4_trace.log.10_t2 /opt/my

How to repeat:
ssh to the node and write "reboot"

[23 Sep 2009 5:43] Jonas Oreland

ndb_error_reporter

[23 Sep 2009 6:05] David Enbal

I've just uploaded ndb_error_report_20090923085331.tar.bz2 to your FTP.

BTW - I just changed the configuration of the cluster to work with only 2 data node (and 2 replicas instead of 3) and everything works like a charm.

[23 Sep 2009 8:08] Bugs System

A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/84256

3032 Jonas Oreland	2009-09-23
      ndb - bug#47530 - 3 (or more) replica with ndbmtd, signal sent to proxy instead of instance

[23 Sep 2009 8:10] Jonas Oreland

pushed to 7.0.8 and 7.1

[23 Sep 2009 9:56] Jon Stephens

Documented bugfix in the NDB-7.0.8 changelog as follows:

        When using multi-threade data nodes (ndbmtd) with NoOfReplicas
        set to a value greater than 2, attempting to restart any of the 
        data nodes caused a forced shutdown of the entire cluster.

Closed.

[30 Sep 2009 8:14] Bugs System

Pushed into 5.1.37-ndb-7.0.9 (revid:[e-mail]-20090930075942-1q6asjcp0gaeynmj) (version source revid:[e-mail]-20090923080856-nieqdjupo5pm6k91) (merge vers: 5.1.37-ndb-7.0.8) (pib:11)

[30 Sep 2009 8:15] Bugs System

Pushed into 5.1.35-ndb-7.1.0 (revid:[e-mail]-20090930080049-1c8a8cio9qgvhq35) (version source revid:[e-mail]-20090923080935-lisreez3v0irakzz) (merge vers: 5.1.35-ndb-7.1.0) (pib:11)