Bug #85674 Unable to perform backup & restore cluster using mcm.
Submitted: 28 Mar 2017 16:13 Modified: 30 Mar 2017 16:49
Reporter: Shay Cohavi Email Updates:
Status: Not a Bug Impact on me:
None 
Category:MySQL Cluster: Configurator Severity:S1 (Critical)
Version:7.4.8 OS:Red Hat (6.6 x86_64)
Assigned to: MySQL Verification Team CPU Architecture:Any
Tags: mcm-1.4.0, ndb-7.4.8 mcm-1.4.0-cluster-7.4.8-linux-rhel5-x86-64bit.rpm

[28 Mar 2017 16:13] Shay Cohavi
Description:
Unable to perform backup & restore procedure using mcm. duo to failure in starting cluster with --initial flag.
servers:
192.168.15.217 ndbd mgmt and mysqld
192.168.15.218 ndbd mgmt and mysqld
192.168.15.219 mysqld

How to repeat:
mcm> start cluster mysql_cluster;
+------------------------------+
| Command result               |
+------------------------------+
| Cluster started successfully |
+------------------------------+
1 row in set (20.06 sec)

mcm> backup cluster mysql_cluster;
+-------------------------------+
| Command result                |
+-------------------------------+
| Backup completed successfully |
+-------------------------------+
1 row in set (1.85 sec)

mcm> list backups mysql_cluster;
+----------+--------+----------------+----------------------+---------+
| BackupId | NodeId | Host           | Timestamp            | Comment |
+----------+--------+----------------+----------------------+---------+
| 1        | 1      | 192.168.15.217 | 2017-03-28 15:57:04Z |         |
| 1        | 2      | 192.168.15.218 | 2017-03-28 15:57:04Z |         |
+----------+--------+----------------+----------------------+---------+
2 rows in set (0.05 sec)

mcm> stop cluster mysql_cluster;
+------------------------------+
| Command result               |
+------------------------------+
| Cluster stopped successfully |
+------------------------------+
1 row in set (18.55 sec)

mcm> show status mysql_cluster;
+---------------+---------+---------+
| Cluster       | Status  | Comment |
+---------------+---------+---------+
| mysql_cluster | stopped |         |
+---------------+---------+---------+
1 row in set (0.04 sec)

mcm> start cluster --initial mysql_cluster;
ERROR 7006 (00MGR): Process error: Node 51 : es to '::';
2017-03-28 18:58:25 9015 [Note] Server socket created on IP: '::'.
2017-03-28 18:58:25 9015 [Note] NDB Index Stat: Starting...
2017-03-28 18:58:25 9015 [Note] NDB Index Stat: Wait for server start completed
2017-03-28 18:58:25 9015 [ERROR] /opt/mcm/cluster/bin/mysqld: Incorrect information in file: './mysql/user.frm'
2017-03-28 18:58:25 9015 [ERROR] Fatal error: Can't open and lock privilege tables: Table 'mysql.user' doesn't exist

mcm> show status -r mysql_cluster;
+--------+----------+----------------+---------+-----------+---------+
| NodeId | Process  | Host           | Status  | Nodegroup | Package |
+--------+----------+----------------+---------+-----------+---------+
| 49     | ndb_mgmd | 192.168.15.217 | running |           | 7.4.8   |
| 50     | ndb_mgmd | 192.168.15.218 | running |           | 7.4.8   |
| 1      | ndbd     | 192.168.15.217 | running | 0         | 7.4.8   |
| 2      | ndbd     | 192.168.15.218 | running | 0         | 7.4.8   |
| 51     | mysqld   | 192.168.15.217 | failed  |           | 7.4.8   |
| 52     | mysqld   | 192.168.15.218 | failed  |           | 7.4.8   |
| 53     | mysqld   | 192.168.15.219 | failed  |           | 7.4.8   |
| 54     | ndbapi   | *              | added   |           |         |
+--------+----------+----------------+---------+-----------+---------+
8 rows in set (0.08 sec)

mcm> restore cluster --backupid=1 mysql_cluster;
ERROR 5200 (00MGR): Restore can not be performed with current process states in cluster mysql_cluster
[28 Mar 2017 16:14] Shay Cohavi
mcmd.log

Attachment: init_mcmd.tar.bz2 (application/x-bzip2, text), 160.44 KiB.

[30 Mar 2017 16:49] MySQL Verification Team
Hi,
Your cluster is in "bad state" so you cannot start a backup from MCM. This is expected behavior. 

Now, "how did you get to this state" might be a bug but the inability to perform backup is not. You might want to create a new bug report about "starting cluster with --initial fails" with all the details on how did you create the cluster.

thanks
Bogdan