Bug #85713 starting cluster with --initial fails
Submitted: 30 Mar 2017 17:12 Modified: 4 Jul 2017 19:59
Reporter: Shay Cohavi Email Updates:
Status: Closed Impact on me:
None 
Category:MySQL Cluster Manager Severity:S1 (Critical)
Version:7.4.8 OS:Red Hat (6.6 x86_64)
Assigned to: CPU Architecture:Any
Tags: mcm-1.4.0, mcm-1.4.0-cluster-7.4.8-linux-rhel5-x86-64bit.rpm, ndb-7.4.8

[30 Mar 2017 17:12] Shay Cohavi
Description:
Unable to perform backup & restore procedure using mcm. 
duo to failure in starting cluster with --initial flag.
servers:
192.168.15.217 ndbd mgmt and mysqld
192.168.15.218 ndbd mgmt and mysqld
192.168.15.219 mysqld

How to repeat:
mcm> start cluster mysql_cluster;
+------------------------------+
| Command result               |
+------------------------------+
| Cluster started successfully |
+------------------------------+
1 row in set (20.06 sec)

mcm> backup cluster mysql_cluster;
+-------------------------------+
| Command result                |
+-------------------------------+
| Backup completed successfully |
+-------------------------------+
1 row in set (1.85 sec)

mcm> list backups mysql_cluster;
+----------+--------+----------------+----------------------+---------+
| BackupId | NodeId | Host           | Timestamp            | Comment |
+----------+--------+----------------+----------------------+---------+
| 1        | 1      | 192.168.15.217 | 2017-03-28 15:57:04Z |         |
| 1        | 2      | 192.168.15.218 | 2017-03-28 15:57:04Z |         |
+----------+--------+----------------+----------------------+---------+
2 rows in set (0.05 sec)

mcm> stop cluster mysql_cluster;
+------------------------------+
| Command result               |
+------------------------------+
| Cluster stopped successfully |
+------------------------------+
1 row in set (18.55 sec)

mcm> show status mysql_cluster;
+---------------+---------+---------+
| Cluster       | Status  | Comment |
+---------------+---------+---------+
| mysql_cluster | stopped |         |
+---------------+---------+---------+
1 row in set (0.04 sec)

mcm> start cluster --initial mysql_cluster;
ERROR 7006 (00MGR): Process error: Node 51 : es to '::';
2017-03-28 18:58:25 9015 [Note] Server socket created on IP: '::'.
2017-03-28 18:58:25 9015 [Note] NDB Index Stat: Starting...
2017-03-28 18:58:25 9015 [Note] NDB Index Stat: Wait for server start completed
2017-03-28 18:58:25 9015 [ERROR] /opt/mcm/cluster/bin/mysqld: Incorrect information in file: './mysql/user.frm'
2017-03-28 18:58:25 9015 [ERROR] Fatal error: Can't open and lock privilege tables: Table 'mysql.user' doesn't exist

mcm> show status -r mysql_cluster;
+--------+----------+----------------+---------+-----------+---------+
| NodeId | Process  | Host           | Status  | Nodegroup | Package |
+--------+----------+----------------+---------+-----------+---------+
| 49     | ndb_mgmd | 192.168.15.217 | running |           | 7.4.8   |
| 50     | ndb_mgmd | 192.168.15.218 | running |           | 7.4.8   |
| 1      | ndbd     | 192.168.15.217 | running | 0         | 7.4.8   |
| 2      | ndbd     | 192.168.15.218 | running | 0         | 7.4.8   |
| 51     | mysqld   | 192.168.15.217 | failed  |           | 7.4.8   |
| 52     | mysqld   | 192.168.15.218 | failed  |           | 7.4.8   |
| 53     | mysqld   | 192.168.15.219 | failed  |           | 7.4.8   |
| 54     | ndbapi   | *              | added   |           |         |
+--------+----------+----------------+---------+-----------+---------+
8 rows in set (0.08 sec)

mcm> restore cluster --backupid=1 mysql_cluster;
ERROR 5200 (00MGR): Restore can not be performed with current process states in cluster mysql_cluster
[30 Mar 2017 17:13] Shay Cohavi
mcmd.log

Attachment: init_mcmd.tar.bz2 (application/x-bzip2, text), 160.44 KiB.

[7 Apr 2017 3:22] MySQL Verification Team
Hi,

Thanks for your bug report, looks like something goes wrong on initial start of mysqld. 

kind regards
Bogdan
[4 Jul 2017 19:59] Daniel So
Posted by developer:
 
Added the following entry to the MCM 1.4.3 changelog:

"The start cluster --initial command now reinitializes the SQL nodes (if their data directories are empty) as well as the data nodes of an NDB Cluster. A new option, --skip-init, has been introduced, for specifying a comma-separated list of the SQL nodes for which reinitialization is to be skipped."

Description for the new option has also been added to the manual.