MySQL Bugs: #43045: mysqld hang/core/... if cluster disconnect during startup of binlog

Bug #43045	mysqld hang/core/... if cluster disconnect during startup of binlog
Submitted:	20 Feb 2009 8:30	Modified:	20 Feb 2009 15:04
Reporter:	Tomas Ulin	Email Updates:
Status:	Closed	Impact on me:	None
Category:	MySQL Cluster: Replication	Severity:	S3 (Non-critical)
Version:	Cluster 6.3	OS:	Any
Assigned to:	Tomas Ulin	CPU Architecture:	Any

Description:
during mysqld startup tables are discovered from cluster and subscriptions are setup for the binlog...

090205 18:42:54 [Note] NDB Binlog: DISCOVER TABLE Event: REPL$test/t1
090205 18:42:54 [Note] NDB Binlog: logging ./test/t1 (UPDATED,USE_WRITE)
090205 18:42:54 [Note] NDB Binlog: DISCOVER TABLE Event: REPL$test/t2
090205 18:42:54 [Note] NDB Binlog: logging ./test/t2 (UPDATED,USE_WRITE)
...

Now if cluster goes down for any reason during this... the mysqld will be in a bad state... and upon reconnect to cluster it may hang/core/...

How to repeat:
.

A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/66987

2867 Tomas Ulin	2009-02-20
      Bug #43045 mysqld hang/core/... if cluster disconnect during startup of binlog
      modified:
        sql/ha_ndbcluster_binlog.cc

Pushed into 5.1.32-ndb-6.3.23 (revid:tomas.ulin@sun.com-20090220084026-ezl8uf2f0rkpzl90) (version source revid:tomas.ulin@sun.com-20090220083906-svdmkd4mm50n61mb) (merge vers: 5.1.32-ndb-6.3.23) (pib:6)

typical printout showing cluster disconnect

090213  1:35:13 [Warning] NDB: Releasing global schema lock (286)Node failure caused abort of transaction
090213  1:35:13 [Note] table './test/t1' opened read only
090213  1:35:22 [Warning] NDB: Could not acquire global schema lock (4009)Cluster Failure
090213  1:35:22 [Warning] NDB: Releasing global schema lock (4025)Node failure caused abort of transaction

Pushed into 5.1.32-ndb-6.4.3 (revid:jonas@mysql.com-20090220085344-ymjrn5kc4rgi3b15) (version source revid:jonas@mysql.com-20090220085344-ymjrn5kc4rgi3b15) (merge vers: 5.1.32-ndb-6.4.3) (pib:6)

Documented bugfix in the NDB-6.3.23 and 6.4.3 changelogs as follows:

        Being disconnected from the cluster while setting up the binary
        log caused mysqld to hang or crash.

I believe this recently affected a customer of mine.  Same error in mysqld log. Restarted mysqld on both API nodes and things working again.

CONFIG on RHEL 5.4:

ndb_mgm> show
Connected to Management Server at: 10.5.102.2:1186
Cluster Configuration
---------------------
[ndbd(NDB)]     2 node(s)
id=2    @10.5.102.4  (mysql-5.1.39 ndb-7.0.9, Nodegroup: 0, Master)
id=3    @10.5.102.5  (mysql-5.1.39 ndb-7.0.9, Nodegroup: 0)

[ndb_mgmd(MGM)] 1 node(s)
id=1    @10.5.102.2  (mysql-5.1.39 ndb-7.0.9)

[mysqld(API)]   2 node(s)
id=4    @10.5.102.2  (mysql-5.1.39 ndb-7.0.9)
id=5    @10.5.102.3  (mysql-5.1.39 ndb-7.0.9)

Could this possibly be marked as S1 or S2?  This is a critical issue and created a loss of service for my customer.