Bug #52859 Replication fails to start after an SQL node restart
Submitted: 15 Apr 2010 14:42 Modified: 12 May 2010 10:20
Reporter: Carlos Pinera Email Updates:
Status: Closed Impact on me:
None 
Category:MySQL Cluster: Replication Severity:S2 (Serious)
Version:mysql-5.1-telco-7.1 OS:Linux (RedHad 5.4 and CentOS)
Assigned to: Martin Skold CPU Architecture:Any
Tags: 7.1.3, replication, sql node restart, telco-7.1 bzr
Triage: Triaged: D2 (Serious) / R6 (Needs Assessment) / E6 (Needs Assessment)

[15 Apr 2010 14:42] Carlos Pinera
Description:
We have multi-master replication between two machines.  Initially the slave status shows everything is fine.  After an SQL node is restarted (/etc/init.d/mysql restart), the slave on that SQL node get an error:

100413 10:15:36 [ERROR] Error reading master configuration
100413 10:15:36 [ERROR] Failed to initialize the master info structure

This was working fine with cluster version 7.0.9 and earlier.  Setting an explicit entry in my.cnf for master-info-file has no effect.  For simplicity I reduced the configuration to two machines, with each machine running a single storage node.  It fails the same way with the configuration of two macnines in each cluster and replication between the clusters (total of 4 machines).  After the restart the storage and sql nodes are up but the slave has not started.

The my.cnf file content for 'andorinha' is:
# MySQL Cluster configuation file. 
[mysqld]
ndbcluster   # Run the NDB storage engine
ndb-connectstring=andorinha.isxinc.com

server_id=3
auto_increment_increment=10
auto_increment_offset=3
binlog-format=ROW
log-bin=row
binlog-ignore-db=test
ndb-log-update-as-write=OFF
log-slave-updates
replicate-same-server-id=0
replicate-ignore-db=test
replicate-do-table=mysql.ndb_apply_status
replicate-wild-ignore-table=mysql.%
relay-log=/var/lib/mysql/andorinha.isxinc.com_rl
relay-log-index=/var/lib/mysql/andorinha.isxinc.com_rl1
relay-log-info-file=/var/lib/mysql/relay-log.info
slave-net-timeout=60

[mysql_cluster]
ndb-connectstring=andorinha.isxinc.com

The my.cnf file content for 'canario' is:
# MySQL Cluster configuation file.
[mysqld]
ndbcluster   # Run the NDB storage engine
ndb-connectstring=canario.isxinc.com

server_id=1
auto_increment_increment=10
auto_increment_offset=1
binlog-format=ROW
log-bin=row
binlog-ignore-db=test
ndb-log-update-as-write=OFF
log-slave-updates
replicate-same-server-id=0
replicate-ignore-db=test
replicate-do-table=mysql.ndb_apply_status
replicate-wild-ignore-table=mysql.%
relay-log=/var/lib/mysql/canario.isxinc.com_rl
relay-log-index=/var/lib/mysql/canario.isxinc.com_rl1
relay-log-info-file=/var/lib/mysql/relay-log.info
slave-net-timeout=60

[mysql_cluster]
ndb-connectstring=canario.isxinc.com
===============================================
The config.ini file (on 'andorinha') :
[tcp default]
SendBufferMemory=2M
ReceiveBufferMemory=2M

[ndbd default]
NoOfReplicas=1
MaxNoOfConcurrentOperations=100000
TimeBetweenGlobalCheckpoints=1000
TimeBetweenEpochs=200
DiskCheckpointSpeed=10M
DiskCheckpointSpeedInRestart=100M
RedoBuffer=32M
DataMemory=427M
IndexMemory=85M

[ndb_mgmd]
HostName=andorinha.isxinc.com
datadir=/var/lib/mysql-cluster

[ndbd]
HostName=andorinha.isxinc.com
datadir=/usr/local/mysql/data
[mysqld]

==============================================

The error messages from the slave error file after the restart:
...
100413 13:53:45 mysqld_safe mysqld from pid file /var/lib/mysql/andorinha.isxinc.com.pid ended
100413 13:53:45 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
100413 13:53:45 [Note] Plugin 'FEDERATED' is disabled.
100413 13:53:45  InnoDB: Started; log sequence number 0 44233
100413 13:53:45 [Note] NDB: NodeID is 3, management server 'andorinha.isxinc.com:1186'
100413 13:53:46 [Note] NDB[0]: NodeID: 3, no storage nodes connected (timed out)
100413 13:53:46 [Note] Starting Cluster Binlog Thread
100413 13:53:46 [ERROR] Error reading master configuration
100413 13:53:46 [ERROR] Failed to initialize the master info structure
100413 13:53:46 [Note] Event Scheduler: Loaded 0 events
100413 13:53:47 [Note] NDB Binlog: Ndb tables initially read only.
100413 13:53:47 [Note] NDB Binlog: DISCOVER TABLE Event: REPL$mysql/ndb_schema
100413 13:53:47 [Note] NDB Binlog: logging ./mysql/ndb_schema (UPDATED,USE_UPDATE)
100413 13:53:47 [Note] NDB Binlog: DISCOVER TABLE Event: REPL$mysql/ndb_apply_status
100413 13:53:47 [Note] NDB Binlog: logging ./mysql/ndb_apply_status (UPDATED,USE_UPDATE)
2010-04-13 13:53:47 [NdbApi] INFO     -- Flushing incomplete GCI:s < 199/0
2010-04-13 13:53:47 [NdbApi] INFO     -- Flushing incomplete GCI:s < 199/0
100413 13:53:47 [Note] NDB Binlog: starting log at epoch 199/0
100413 13:53:47 [Note] NDB Binlog: ndb tables writable
100413 13:53:47 [Note] /usr/sbin/mysqld: ready for connections.
Version: '5.1.44-ndb-7.1.3-cluster-gpl-log'  socket: '/var/lib/mysql/mysql.sock'  port: 3306  MySQL Cluster Server (GPL)

The output of SHOW SLAVE STATUS after the restart
*************************** 1. row ***************************
               Slave_IO_State:
                  Master_Host: canario.isxinc.com
                  Master_User: repl
                  Master_Port: 0
                Connect_Retry: 0
              Master_Log_File: row.000001
          Read_Master_Log_Pos: 0
               Relay_Log_File:
                Relay_Log_Pos: 0
        Relay_Master_Log_File:
             Slave_IO_Running: No
            Slave_SQL_Running: No
              Replicate_Do_DB:
          Replicate_Ignore_DB: test
           Replicate_Do_Table: mysql.ndb_apply_status
       Replicate_Ignore_Table:
      Replicate_Wild_Do_Table:
  Replicate_Wild_Ignore_Table: mysql.%
                   Last_Errno: 0
                   Last_Error:
                 Skip_Counter: 0
          Exec_Master_Log_Pos: 0
              Relay_Log_Space: 0
              Until_Condition: None
               Until_Log_File:
                Until_Log_Pos: 0
           Master_SSL_Allowed: No
           Master_SSL_CA_File:
           Master_SSL_CA_Path:
              Master_SSL_Cert:
            Master_SSL_Cipher:
               Master_SSL_Key:
        Seconds_Behind_Master: NULL
Master_SSL_Verify_Server_Cert: No
                Last_IO_Errno: 0
                Last_IO_Error:
               Last_SQL_Errno: 0
               Last_SQL_Error:
                  Master_Bind:
  Replicate_Ignore_Server_Ids:
             Master_Server_Id: 0

How to repeat:
A single master/slave setup is sufficient.  Do the 'change master' on the slave, and start the slave.  Perform SHOW SLAVE STATUS \G; and verify that "Slave_IO_Running" and "Slave_SQL_Running" are both "YES".

Restart the sql node where the slave is running, for example "/etc/init.d/mysql restart".  Perform SHOW SLAVE STATUS \G; and see that "Slave_IO_Running" and "Slave_SQL_Running" are both "NO" and the error file in /var/lib/mysql shows the two error messages listed in the description section.

Retry this on version 7.0.9 and earlier and the error does not occur.
[16 Apr 2010 7:38] Sveta Smirnova
Thank you for the report.

Verified as described.
[16 Apr 2010 13:02] Tom Farvour
We are having the same bug with 7.1.3 (MySQL 5.1.44), and also was having the problem with 7.0.13 (MySQL 5.1.41) as described. It wasn't an issue in 7.0.9 (MySQL 5.1.39)

Please fix.
[27 Apr 2010 9:10] Valenti Jove
We have the same bug (mysql cluster 7.1.3).

Replication is running ok. We do a mysql stop and mysql start, and replication doesn't work anymore.

>start slave;
ERROR 1201 (HY000): Could not initialize master info structure; more error messages can be found in the MySQL error log

>show slave status;
+----------------+--------------+-------------+-------------+---------------+------------------+---------------------+----------------+---------------+-----------------------+------------------+-------------------+-----------------+---------------------+--------------------+------------------------+-------------------------+-----------------------------+------------+------------+--------------+---------------------+-----------------+-----------------+----------------+---------------+--------------------+--------------------+--------------------+-----------------+-------------------+----------------+-----------------------+-------------------------------+---------------+---------------+----------------+----------------+-------------+-----------------------------+------------------+
| Slave_IO_State | Master_Host  | Master_User | Master_Port | Connect_Retry | Master_Log_File  | Read_Master_Log_Pos | Relay_Log_File | Relay_Log_Pos | Relay_Master_Log_File | Slave_IO_Running | Slave_SQL_Running | Replicate_Do_DB | Replicate_Ignore_DB | Replicate_Do_Table | Replicate_Ignore_Table | Replicate_Wild_Do_Table | Replicate_Wild_Ignore_Table | Last_Errno | Last_Error | Skip_Counter | Exec_Master_Log_Pos | Relay_Log_Space | Until_Condition | Until_Log_File | Until_Log_Pos | Master_SSL_Allowed | Master_SSL_CA_File | Master_SSL_CA_Path | Master_SSL_Cert | Master_SSL_Cipher | Master_SSL_Key | Seconds_Behind_Master | Master_SSL_Verify_Server_Cert | Last_IO_Errno | Last_IO_Error | Last_SQL_Errno | Last_SQL_Error | Master_Bind | Replicate_Ignore_Server_Ids | Master_Server_Id |
+----------------+--------------+-------------+-------------+---------------+------------------+---------------------+----------------+---------------+-----------------------+------------------+-------------------+-----------------+---------------------+--------------------+------------------------+-------------------------+-----------------------------+------------+------------+--------------+---------------------+-----------------+-----------------+----------------+---------------+--------------------+--------------------+--------------------+-----------------+-------------------+----------------+-----------------------+-------------------------------+---------------+---------------+----------------+----------------+-------------+-----------------------------+------------------+
|                | 192.168.10.7 | repl        |           0 |             0 | mysql-bin.001320 |                   0 |                |             0 |                       | No               | No                | db   |                     |                    |                        | db.%         |                             |          0 |            |            0 |                   0 |               0 | None            |                |             0 | No                 |                    |                    |                 |                   |                |                  NULL | No                            |             0 |               |              0 |                |             |                             |                0 |
+----------------+--------------+-------------+-------------+---------------+------------------+---------------------+----------------+---------------+-----------------------+------------------+-------------------+-----------------+---------------------+--------------------+------------------------+-------------------------+-----------------------------+------------+------------+--------------+---------------------+-----------------+-----------------+----------------+---------------+--------------------+--------------------+--------------------+-----------------+-------------------+----------------+-----------------------+-------------------------------+---------------+---------------+----------------+----------------+-------------+-----------------------------+------------------+
[5 May 2010 11:24] Martin Skold
Seems the master.info file on slave had two lines swapped, making patch and adding new test case.
[6 May 2010 12:16] Bugs System
A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/107634

3192 Martin Skold	2010-05-06
      Bug#52859 Replication fails to start after an SQL node restart: Lines for heartbeat period and bind addr were swapped in master.info
      modified:
        sql/rpl_mi.cc
[6 May 2010 12:18] Bugs System
A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/107635

3193 Martin Skold	2010-05-06
      Bug#52859 Replication fails to start after an SQL node restart: Added new test for restarting slave
      added:
        mysql-test/suite/rpl_ndb/r/rpl_ndb_slave_restart.result
        mysql-test/suite/rpl_ndb/t/rpl_ndb_slave_restart.test
[6 May 2010 12:28] Bugs System
A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/107637

3525 Martin Skold	2010-05-06 [merge]
      Merge
      added:
        mysql-test/suite/rpl_ndb/r/rpl_ndb_slave_restart.result
        mysql-test/suite/rpl_ndb/t/rpl_ndb_slave_restart.test
      modified:
        sql/rpl_mi.cc
[6 May 2010 12:31] Bugs System
A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/107638

3594 Martin Skold	2010-05-06 [merge]
      Merge
      added:
        mysql-test/suite/rpl_ndb/r/rpl_ndb_slave_restart.result
        mysql-test/suite/rpl_ndb/t/rpl_ndb_slave_restart.test
      modified:
        sql/rpl_mi.cc
[6 May 2010 12:38] Bugs System
Pushed into 5.1.44-ndb-7.0.15 (revid:martin.skold@mysql.com-20100506123735-3zd8rkua1ycl569c) (version source revid:martin.skold@mysql.com-20100506122822-2hntg1bkhkxxsh1m) (merge vers: 5.1.44-ndb-7.0.15) (pib:16)
[6 May 2010 12:41] Bugs System
Pushed into 5.1.44-ndb-6.3.34 (revid:martin.skold@mysql.com-20100506124030-01g0x6mi14sbb92g) (version source revid:martin.skold@mysql.com-20100506121756-37ny8k4n3fmft5oq) (merge vers: 5.1.44-ndb-6.3.34) (pib:16)
[10 May 2010 6:25] Bugs System
A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/107816

3529 Martin Skold	2010-05-10
      Bug#52859 Replication fails to start after an SQL node restart: Fixed test case to work in Windows
      modified:
        mysql-test/suite/rpl_ndb/r/rpl_ndb_slave_restart.result
        mysql-test/suite/rpl_ndb/t/rpl_ndb_slave_restart.test
[10 May 2010 6:27] Bugs System
A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/107817

3600 Martin Skold	2010-05-10 [merge]
      Merge
      modified:
        mysql-test/suite/rpl_ndb/r/rpl_ndb_slave_restart.result
        mysql-test/suite/rpl_ndb/t/rpl_ndb_slave_restart.test
[12 May 2010 10:20] Jon Stephens
Documented bugfix in the NDB-6.3.34, 7.0.15, and 7.1.4 changelogs, as follows:

      Replication failed after a restart of the slave SQL node,
      due to an error in writing the master.info file.

Closed.