MySQL Bugs: #112929: InnoDB ClusterSet Replica Cluster creation fails due to account creation failure

Bug #112929	InnoDB ClusterSet Replica Cluster creation fails due to account creation failure
Submitted:	1 Nov 2023 17:10	Modified:	10 Nov 2023 19:13
Reporter:	Hamza Ahmed	Email Updates:
Status:	Can't repeat	Impact on me:	None
Category:	Shell AdminAPI InnoDB Cluster / ReplicaSet	Severity:	S3 (Non-critical)
Version:	8.0.30, 8.0.32	OS:	CentOS
Assigned to:	MySQL Verification Team	CPU Architecture:	Any
Tags:	InnoDB ClusterSet, mysqlshell

Description:
I am trying to create a new replica cluster in an InnoDB ClusterSet, but the process fails with an error as below:
ERROR: Error creating Replica Cluster: Unable to create the Group Replication recovery account: current_clsset_primary_HOST:PORT: You are not allowed to create a user with GRANT 

We upgraded from 8.0.30 to 8.0.32 because we thought this was related to the bugs reported on https://support.oracle.com/epmos/faces/DocContentDisplay?id=2904745.1, but that has not helped.

The flow looks like below:
```
[root@snc-dbmytfmi601 ~]# mysqlsh -uicadmin -p --quiet-start=2
 MySQL  localhost:33060+ ssl  JS > clsset = dba.getClusterSet()
<ClusterSet:clsset_name>
 MySQL  localhost:33060+ ssl  JS > clsset.createReplicaCluster("icadmin@REPLICA_CLUSTER_PRIMARY_HOST:PORT", "cluster2")
Setting up replica 'cluster2' of cluster 'cluster1' at instance 'REPLICA_CLUSTER_PRIMARY_HOST:PORT'.

A new InnoDB Cluster will be created on instance 'REPLICA_CLUSTER_PRIMARY_HOST:PORT'.

Validating instance configuration at REPLICA_CLUSTER_PRIMARY_HOST:PORT...

This instance reports its own address as REPLICA_CLUSTER_PRIMARY_HOST:PORT

Instance configuration is suitable.
NOTE: Group Replication will communicate with other members using 'REPLICA_CLUSTER_PRIMARY_HOST:PORT'. Use the localAddress option to override.

* Checking connectivity and SSL configuration...

* Checking transaction state of the instance...
The safest and most convenient way to provision a new instance is through automatic clone provisioning, which will completely overwrite the state of 'REPLICA_CLUSTER_PRIMARY_HOST:PORT' with a physical snapshot from an existing clusterset member. To use this method by default, set the 'recoveryMethod' option to 'clone'.

WARNING: It should be safe to rely on replication to incrementally recover the state of the new Replica Cluster if you are sure all updates ever executed in the ClusterSet were done with GTIDs enabled, there are no purged transactions and the instance used to create the new Replica Cluster contains the same GTID set as the ClusterSet or a subset of it. To use this method by default, set the 'recoveryMethod' option to 'incremental'.

Incremental state recovery was selected because it seems to be safely usable.

* Checking connectivity and SSL configuration to PRIMARY Cluster...

ERROR: Error creating Replica Cluster: Unable to create the Group Replication recovery account: current_clsset_primary_HOST:PORT: You are not allowed to create a user with GRANT
NOTE: Reverting changes...

Changes successfully reverted.
ClusterSet.createReplicaCluster: Unable to create the Group Replication recovery account: current_clsset_primary_HOST:PORT: You are not allowed to create a user with GRANT (RuntimeError)
 MySQL  localhost:33060+ ssl  JS >
```

The grants for our icadmin don't seem to have any permission issues:
```
mysql> show grants for 'icadmin'@'%'\G
*************************** 1. row ***************************
Grants for icadmin@%: GRANT SELECT, INSERT, UPDATE, DELETE, CREATE, DROP, RELOAD, SHUTDOWN, PROCESS, FILE, REFERENCES, INDEX, ALTER, SHOW DATABASES, SUPER, CREATE TEMPORARY TABLES, LOCK TABLES, EXECUTE, REPLICATION SLAVE, REPLICATION CLIENT, CREATE VIEW, SHOW VIEW, CREATE ROUTINE, ALTER ROUTINE, CREATE USER, EVENT, TRIGGER, CREATE TABLESPACE, CREATE ROLE, DROP ROLE ON *.* TO `icadmin`@`%` WITH GRANT OPTION
*************************** 2. row ***************************
Grants for icadmin@%: GRANT APPLICATION_PASSWORD_ADMIN,AUDIT_ABORT_EXEMPT,AUDIT_ADMIN,AUTHENTICATION_POLICY_ADMIN,BACKUP_ADMIN,BINLOG_ADMIN,BINLOG_ENCRYPTION_ADMIN,CLONE_ADMIN,CONNECTION_ADMIN,ENCRYPTION_KEY_ADMIN,FIREWALL_EXEMPT,FLUSH_OPTIMIZER_COSTS,FLUSH_STATUS,FLUSH_TABLES,FLUSH_USER_RESOURCES,GROUP_REPLICATION_ADMIN,GROUP_REPLICATION_STREAM,INNODB_REDO_LOG_ARCHIVE,INNODB_REDO_LOG_ENABLE,PASSWORDLESS_USER_ADMIN,PERSIST_RO_VARIABLES_ADMIN,REPLICATION_APPLIER,REPLICATION_SLAVE_ADMIN,RESOURCE_GROUP_ADMIN,RESOURCE_GROUP_USER,ROLE_ADMIN,SENSITIVE_VARIABLES_OBSERVER,SERVICE_CONNECTION_ADMIN,SESSION_VARIABLES_ADMIN,SET_USER_ID,SHOW_ROUTINE,SYSTEM_USER,SYSTEM_VARIABLES_ADMIN,TABLE_ENCRYPTION_ADMIN,XA_RECOVER_ADMIN ON *.* TO `icadmin`@`%` WITH GRANT OPTION
*************************** 3. row ***************************
Grants for icadmin@%: GRANT INSERT, UPDATE, DELETE ON `mysql`.* TO `icadmin`@`%` WITH GRANT OPTION
*************************** 4. row ***************************
Grants for icadmin@%: GRANT INSERT, UPDATE, DELETE, CREATE, DROP, REFERENCES, INDEX, ALTER, CREATE TEMPORARY TABLES, LOCK TABLES, EXECUTE, CREATE VIEW, SHOW VIEW, CREATE ROUTINE, ALTER ROUTINE, EVENT, TRIGGER ON `mysql_innodb_cluster_metadata_bkp`.* TO `icadmin`@`%` WITH GRANT OPTION
4 rows in set (0.01 sec)

mysql> ^DBye
```

How to repeat:
1. Connect to mysqlsh primary via the cluster/clusterset's admin credentials: `mysqlsh -uicadmin -p --quiet-start=2`
2. Set the clsset object in mysqlsh: `clsset = dba.getClusterSet()`
3. Try creating the replica cluster: `clsset.createReplicaCluster("icadmin@REPLICA_CLUSTER_PRIMARY_HOST:PORT", "cluster2")`

How was the instance being used for the replica cluster setup? What server version is being used?

Initially I was letting mysqlsh perform the cloning during the replica cluster creation, but for this attempt I performed the cloning beforehand with the clone plugin.

mysqlsh is on 8.0.35, but all other mysql binaries are on 8.0.32. We were running into the same issue when everything was on 8.0.30.

For some reason, the list of grants for your icadmin user is missing 

GRANT INSERT, UPDATE, DELETE, CREATE, DROP, REFERENCES, INDEX, ALTER, CREATE TEMPORARY TABLES, LOCK TABLES, EXECUTE, CREATE VIEW, SHOW VIEW, CREATE ROUTINE, ALTER ROUTINE, EVENT, TRIGGER ON `mysql_innodb_cluster_metadata_previous`.* TO `icadmin`@`%` WITH GRANT OPTION

How was the icadmin account created and with what version of the shell?

Also, to see exactly what statement is failing, please try createReplicaCluster again while the shell is started with: --verbose=1 --log-sql=all

> For some reason, the list of grants for your icadmin user is missing 
> 
> GRANT INSERT, UPDATE, DELETE, CREATE, DROP, REFERENCES, INDEX, ALTER, CREATE TEMPORARY TABLES, LOCK TABLES, EXECUTE, CREATE VIEW, SHOW VIEW, CREATE ROUTINE, ALTER ROUTINE, EVENT, TRIGGER ON `mysql_innodb_cluster_metadata_previous`.* TO `icadmin`@`%` WITH GRANT OPTION

Should I manually run that and try to add the privileges?

> How was the icadmin account created and with what version of the shell?

The account is created manually with the statement: GRANT ALL ON *.* to 'icadmin'@'%' with grant option;"
Not certain, but likely some version between 8.0.24-8.0.28. Would it matter?

> Also, to see exactly what statement is failing, please try createReplicaCluster again while the shell is started with: --verbose=1 --log-sql=all

Failure seems to be on a grant statement:

verbose: 2023-11-07T07:43:30Z: Creating replica cluster on 'REPLICA_CLUSTER_PRIMARY_HOST:PORT'
verbose: 2023-11-07T07:43:30Z: ClusterSet.createReplicaCluster: tid=2296802: SQL: SELECT attributes->'$.opt_clusterSetReplicationSslMode' FROM mysql_innodb_cluster_metadata.clustersets WHERE clusterset_id='9984ad61-5b57-11ee-a68d-0dac3a1bdbfa'
verbose: 2023-11-07T07:43:30Z: ClusterSet.createReplicaCluster: tid=2296802: SQL: SELECT attributes->'$.opt_memberAuthType' FROM mysql_innodb_cluster_metadata.clustersets WHERE clusterset_id='9984ad61-5b57-11ee-a68d-0dac3a1bdbfa'
verbose: 2023-11-07T07:43:30Z: ClusterSet.createReplicaCluster: tid=2296802: SQL: SELECT attributes->'$.opt_memberAuthType' FROM mysql_innodb_cluster_metadata.clusters WHERE cluster_id='03e50357-5b50-11ee-a68d-0dac3a1bdbfa'
verbose: 2023-11-07T07:43:30Z: ClusterSet.createReplicaCluster: tid=2296802: SQL: SELECT attributes->'$.opt_certIssuer' FROM mysql_innodb_cluster_metadata.clusters WHERE cluster_id='03e50357-5b50-11ee-a68d-0dac3a1bdbfa'
verbose: 2023-11-07T07:43:30Z: ClusterSet.createReplicaCluster: tid=13631: SQL: SELECT @@server_id
verbose: 2023-11-07T07:43:30Z: ClusterSet.createReplicaCluster: tid=2296802: SQL: SELECT attributes->'$.opt_replicationAllowedHost' FROM mysql_innodb_cluster_metadata.clustersets WHERE clusterset_id='9984ad61-5b57-11ee-a68d-0dac3a1bdbfa'
verbose: 2023-11-07T07:43:30Z: Creating async replication account 'mysql_innodb_cs_e11'@'%' for new cluster at REPLICA_CLUSTER_PRIMARY_HOST:PORT
verbose: 2023-11-07T07:43:30Z: ClusterSet.createReplicaCluster: tid=2296802: SQL: SELECT concat(quote(user), '@', quote(host)) from mysql.user where user='mysql_innodb_cs_e11'
verbose: 2023-11-07T07:43:30Z: ClusterSet.createReplicaCluster: tid=2296802: SQL: GRANT REPLICATION SLAVE, BACKUP_ADMIN ON *.* TO 'mysql_innodb_cs_e11'@'%'
verbose: 2023-11-07T07:43:30Z: ClusterSet.createReplicaCluster: tid=2296802: MySQL Error 1410 (42000): You are not allowed to create a user with GRANT
ERROR: Error creating Replica Cluster: Unable to create the Group Replication recovery account: current_clsset_primary_HOST:PORT: You are not allowed to create a user with GRANT
NOTE: Reverting changes...

Changes successfully reverted.
verbose: 2023-11-07T07:43:30Z: ClusterSet.createReplicaCluster: tid=13631: SQL: SELECT service_release_locks('AdminAPI_instance')
verbose: 2023-11-07T07:43:30Z: ClusterSet.createReplicaCluster: tid=2296801: SQL: SELECT service_release_locks('AdminAPI_clusterset')
ClusterSet.createReplicaCluster: Unable to create the Group Replication recovery account: current_clsset_primary_HOST:PORT: You are not allowed to create a user with GRANT (RuntimeError)
 MySQL  localhost:33060+ ssl  JS >

Hi Hamza,

>> For some reason, the list of grants for your icadmin user is missing 
>> 
>> GRANT INSERT, UPDATE, DELETE, CREATE, DROP, REFERENCES, INDEX, ALTER, CREATE TEMPORARY TABLES, LOCK TABLES, EXECUTE, CREATE VIEW, SHOW VIEW, CREATE ROUTINE, ALTER ROUTINE, EVENT, TRIGGER ON `mysql_innodb_cluster_metadata_previous`.* TO `icadmin`@`%` WITH GRANT OPTION

> Should I manually run that and try to add the privileges?

Yes.

>> How was the icadmin account created and with what version of the shell?

> The account is created manually with the statement: GRANT ALL ON *.* to 'icadmin'@'%' with grant option;"
> Not certain, but likely some version between 8.0.24-8.0.28. Would it matter?

It doesn't matter. But for reference, you can create an admin account using `Cluster.setupAdminAccount()`

See: https://dev.mysql.com/doc/mysql-shell/8.0/en/creating-user-accounts-for-admin-api.html

That'd create the account with all the strictly necessary grants:

mysqlsh-js> cluster.setupAdminAccount("icadmin");
mysqlsh-js> \sql show grants for 'icadmin@'%'\G

*************************** 1. row ***************************
Grants for icadmin@%: GRANT SELECT, RELOAD, SHUTDOWN, PROCESS, FILE, EXECUTE, REPLICATION SLAVE, REPLICATION CLIENT, CREATE USER ON *.* TO `icadmin`@`%` WITH GRANT OPTION
*************************** 2. row ***************************
Grants for icadmin@%: GRANT CLONE_ADMIN,CONNECTION_ADMIN,GROUP_REPLICATION_ADMIN,PERSIST_RO_VARIABLES_ADMIN,REPLICATION_APPLIER,REPLICATION_SLAVE_ADMIN,ROLE_ADMIN,SYSTEM_VARIABLES_ADMIN ON *.* TO `icadmin`@`%` WITH GRANT OPTION
*************************** 3. row ***************************
Grants for icadmin@%: GRANT INSERT, UPDATE, DELETE ON `mysql`.* TO `icadmin`@`%` WITH GRANT OPTION
*************************** 4. row ***************************
Grants for icadmin@%: GRANT INSERT, UPDATE, DELETE, CREATE, DROP, REFERENCES, INDEX, ALTER, CREATE TEMPORARY TABLES, LOCK TABLES, EXECUTE, CREATE VIEW, SHOW VIEW, CREATE ROUTINE, ALTER ROUTINE, EVENT, TRIGGER ON `mysql_innodb_cluster_metadata`.* TO `icadmin`@`%` WITH GRANT OPTION
*************************** 5. row ***************************
Grants for icadmin@%: GRANT INSERT, UPDATE, DELETE, CREATE, DROP, REFERENCES, INDEX, ALTER, CREATE TEMPORARY TABLES, LOCK TABLES, EXECUTE, CREATE VIEW, SHOW VIEW, CREATE ROUTINE, ALTER ROUTINE, EVENT, TRIGGER ON `mysql_innodb_cluster_metadata_bkp`.* TO `icadmin`@`%` WITH GRANT OPTION
*************************** 6. row ***************************
Grants for icadmin@%: GRANT INSERT, UPDATE, DELETE, CREATE, DROP, REFERENCES, INDEX, ALTER, CREATE TEMPORARY TABLES, LOCK TABLES, EXECUTE, CREATE VIEW, SHOW VIEW, CREATE ROUTINE, ALTER ROUTINE, EVENT, TRIGGER ON `mysql_innodb_cluster_metadata_previous`.* TO `icadmin`@`%` WITH GRANT OPTION
6 rows in set (0.0004 sec)

Just to add to the bug, this can not be reproduced using latest mysql server and mysql shell tools.

I updated the privileges manually, as well as ran an update with setupAdminAccount, so the privileges got updated:
```
 MySQL  localhost:33060+ ssl  JS > \sql  show grants for 'icadmin'\G
verbose: 2023-11-08T07:38:52Z: sql: tid=2601426: SQL:  show grants for 'icadmin'
*************************** 1. row ***************************
Grants for icadmin@%: GRANT SELECT, INSERT, UPDATE, DELETE, CREATE, DROP, RELOAD, SHUTDOWN, PROCESS, FILE, REFERENCES, INDEX, ALTER, SHOW DATABASES, SUPER, CREATE TEMPORARY TABLES, LOCK TABLES, EXECUTE, REPLICATION SLAVE, REPLICATION CLIENT, CREATE VIEW, SHOW VIEW, CREATE ROUTINE, ALTER ROUTINE, CREATE USER, EVENT, TRIGGER, CREATE TABLESPACE, CREATE ROLE, DROP ROLE ON *.* TO `icadmin`@`%` WITH GRANT OPTION
*************************** 2. row ***************************
Grants for icadmin@%: GRANT APPLICATION_PASSWORD_ADMIN,AUDIT_ABORT_EXEMPT,AUDIT_ADMIN,AUTHENTICATION_POLICY_ADMIN,BACKUP_ADMIN,BINLOG_ADMIN,BINLOG_ENCRYPTION_ADMIN,CLONE_ADMIN,CONNECTION_ADMIN,ENCRYPTION_KEY_ADMIN,FIREWALL_EXEMPT,FLUSH_OPTIMIZER_COSTS,FLUSH_STATUS,FLUSH_TABLES,FLUSH_USER_RESOURCES,GROUP_REPLICATION_ADMIN,GROUP_REPLICATION_STREAM,INNODB_REDO_LOG_ARCHIVE,INNODB_REDO_LOG_ENABLE,PASSWORDLESS_USER_ADMIN,PERSIST_RO_VARIABLES_ADMIN,REPLICATION_APPLIER,REPLICATION_SLAVE_ADMIN,RESOURCE_GROUP_ADMIN,RESOURCE_GROUP_USER,ROLE_ADMIN,SENSITIVE_VARIABLES_OBSERVER,SERVICE_CONNECTION_ADMIN,SESSION_VARIABLES_ADMIN,SET_USER_ID,SHOW_ROUTINE,SYSTEM_USER,SYSTEM_VARIABLES_ADMIN,TABLE_ENCRYPTION_ADMIN,XA_RECOVER_ADMIN ON *.* TO `icadmin`@`%` WITH GRANT OPTION
*************************** 3. row ***************************
Grants for icadmin@%: GRANT INSERT, UPDATE, DELETE ON `mysql`.* TO `icadmin`@`%` WITH GRANT OPTION
*************************** 4. row ***************************
Grants for icadmin@%: GRANT INSERT, UPDATE, DELETE, CREATE, DROP, REFERENCES, INDEX, ALTER, CREATE TEMPORARY TABLES, LOCK TABLES, EXECUTE, CREATE VIEW, SHOW VIEW, CREATE ROUTINE, ALTER ROUTINE, EVENT, TRIGGER ON `mysql_innodb_cluster_metadata`.* TO `icadmin`@`%` WITH GRANT OPTION
*************************** 5. row ***************************
Grants for icadmin@%: GRANT INSERT, UPDATE, DELETE, CREATE, DROP, REFERENCES, INDEX, ALTER, CREATE TEMPORARY TABLES, LOCK TABLES, EXECUTE, CREATE VIEW, SHOW VIEW, CREATE ROUTINE, ALTER ROUTINE, EVENT, TRIGGER ON `mysql_innodb_cluster_metadata_bkp`.* TO `icadmin`@`%` WITH GRANT OPTION
*************************** 6. row ***************************
Grants for icadmin@%: GRANT INSERT, UPDATE, DELETE, CREATE, DROP, REFERENCES, INDEX, ALTER, CREATE TEMPORARY TABLES, LOCK TABLES, EXECUTE, CREATE VIEW, SHOW VIEW, CREATE ROUTINE, ALTER ROUTINE, EVENT, TRIGGER ON `mysql_innodb_cluster_metadata_previous`.* TO `icadmin`@`%` WITH GRANT OPTION
6 rows in set (0.0003 sec)
verbose: 2023-11-08T07:38:52Z: main: tid=2601426: SQL: select schema()
 MySQL  localhost:33060+ ssl  JS >
```

But still running into the same failure:
```
verbose: 2023-11-08T07:38:39Z: Creating replica cluster on 'REPLICA_CLUSTER_PRIMARY_HOST:PORT'
verbose: 2023-11-08T07:38:39Z: ClusterSet.createReplicaCluster: tid=2603334: SQL: SELECT attributes->'$.opt_clusterSetReplicationSslMode' FROM mysql_innodb_cluster_metadata.clustersets WHERE clusterset_id='9984ad61-5b57-11ee-a68d-0dac3a1bdbfa'
verbose: 2023-11-08T07:38:39Z: ClusterSet.createReplicaCluster: tid=2603334: SQL: SELECT attributes->'$.opt_memberAuthType' FROM mysql_innodb_cluster_metadata.clustersets WHERE clusterset_id='9984ad61-5b57-11ee-a68d-0dac3a1bdbfa'
verbose: 2023-11-08T07:38:39Z: ClusterSet.createReplicaCluster: tid=2603334: SQL: SELECT attributes->'$.opt_memberAuthType' FROM mysql_innodb_cluster_metadata.clusters WHERE cluster_id='03e50357-5b50-11ee-a68d-0dac3a1bdbfa'
verbose: 2023-11-08T07:38:39Z: ClusterSet.createReplicaCluster: tid=2603334: SQL: SELECT attributes->'$.opt_certIssuer' FROM mysql_innodb_cluster_metadata.clusters WHERE cluster_id='03e50357-5b50-11ee-a68d-0dac3a1bdbfa'
verbose: 2023-11-08T07:38:39Z: ClusterSet.createReplicaCluster: tid=16557: SQL: SELECT @@server_id
verbose: 2023-11-08T07:38:39Z: ClusterSet.createReplicaCluster: tid=2603334: SQL: SELECT attributes->'$.opt_replicationAllowedHost' FROM mysql_innodb_cluster_metadata.clustersets WHERE clusterset_id='9984ad61-5b57-11ee-a68d-0dac3a1bdbfa'
verbose: 2023-11-08T07:38:39Z: Creating async replication account 'mysql_innodb_cs_e11'@'%' for new cluster at REPLICA_CLUSTER_PRIMARY_HOST:PORT
verbose: 2023-11-08T07:38:39Z: ClusterSet.createReplicaCluster: tid=2603334: SQL: SELECT concat(quote(user), '@', quote(host)) from mysql.user where user='mysql_innodb_cs_e11'
verbose: 2023-11-08T07:38:39Z: ClusterSet.createReplicaCluster: tid=2603334: SQL: GRANT REPLICATION SLAVE, BACKUP_ADMIN ON *.* TO 'mysql_innodb_cs_e11'@'%'
verbose: 2023-11-08T07:38:39Z: ClusterSet.createReplicaCluster: tid=2603334: MySQL Error 1410 (42000): You are not allowed to create a user with GRANT
ERROR: Error creating Replica Cluster: Unable to create the Group Replication recovery account: current_clsset_primary_HOST:PORT: You are not allowed to create a user with GRANT
NOTE: Reverting changes...

Changes successfully reverted.
verbose: 2023-11-08T07:38:39Z: ClusterSet.createReplicaCluster: tid=16557: SQL: SELECT service_release_locks('AdminAPI_instance')
verbose: 2023-11-08T07:38:39Z: ClusterSet.createReplicaCluster: tid=2603333: SQL: SELECT service_release_locks('AdminAPI_clusterset')
ClusterSet.createReplicaCluster: Unable to create the Group Replication recovery account: current_clsset_primary_HOST:PORT: You are not allowed to create a user with GRANT (RuntimeError)
 MySQL  localhost:33060+ ssl  JS >
```

We have a lot more environments on 8.0.30, but this is the only one where we are running into this issue, even after we upgraded it to 8.0.32.

I am able to create a user and grant it the same permissions manually without running into any errors, so not sure where the error is stemming from:

```
mysql> CREATE USER 'htest01'@'%' identified by 'badpass1234';
Query OK, 0 rows affected (0.01 sec)

mysql> GRANT REPLICATION SLAVE, BACKUP_ADMIN ON *.* TO 'htest01'@'%' ;
Query OK, 0 rows affected (0.00 sec)

mysql> select user();
+-------------------+
| user()            |
+-------------------+
| icadmin@localhost |
+-------------------+
1 row in set (0.00 sec)

mysql> drop user 'htest01'@'%' ;
Query OK, 0 rows affected (0.00 sec)

mysql> ^DBye
```

I've just noticed that you're attempting to create the Replica Cluster using:

`clsset.createReplicaCluster("icadmin@REPLICA_CLUSTER_PRIMARY_HOST:PORT", "cluster2")`

You're passing a username to the target instance's URI: 'icadmin'. That's fine, although unnecessary since all instances in the topology must have the management account used. So in this case, if you're obtaining the ClusterSet handle (getClusterSet()) when connected to an instance using the account 'icadmin':

[root@snc-dbmytfmi601 ~]# mysqlsh -uicadmin -p --quiet-start=2

... then 'REPLICA_CLUSTER_PRIMARY_HOST:PORT' must have the 'icadmin' account.

It seems it has it, otherwise, the command would fail immediately, however, I wonder whether it has all the needed grants. I'm afraid you were checking the grants for the account at 'snc-dbmytfmi601'. Please double-check.

As I mentioned before, `cluster.setupAdminAccount()` can be used to create an administration account, however, that's the account used when a topology is already configured. In this case, for that account to exist in all the ClusterSet, you'd use `clusterset.setupAdminAccount()`.

A Server Configuration Account must be used for instances that do not belong to the topology *yet*. You set this account up using a dba.configureInstance() with the option 'clusterAdmin', for example: dba.configureInstance("root@REPLICA_CLUSTER_PRIMARY_HOST:PORT", {clusterAdmin: "icadmin"})

For reference: https://dev.mysql.com/doc/mysql-shell/8.0/en/creating-user-accounts-for-admin-api.html

I have tried it both ways - with and without the clusterAdmin name (`icadmin`), but it still fails with the same issue. 

The privileges on the replicaCluster primary are the same. We don't allow remote connectivity with root, but I have logged into the host with the `icadmin` account and rerun dba.configureInstance(), and there are no issues. Additionally, if there were issues with the instance configuration, they would/should have been raised during the initial check (as happened when I completely wiped that instance and let mysqlsh reclone it, multiple times).

The privileges were initially granted with the same `ClusterSet.setupAdminAccount()` command, and even after a rerun across the current ClusterSet as well as the node that I intend to add, they are the same:
```
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         |
+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| GRANT SELECT, INSERT, UPDATE, DELETE, CREATE, DROP, RELOAD, SHUTDOWN, PROCESS, FILE, REFERENCES, INDEX, ALTER, SHOW DATABASES, SUPER, CREATE TEMPORARY TABLES, LOCK TABLES, EXECUTE, REPLICATION SLAVE, REPLICATION CLIENT, CREATE VIEW, SHOW VIEW, CREATE ROUTINE, ALTER ROUTINE, CREATE USER, EVENT, TRIGGER, CREATE TABLESPACE, CREATE ROLE, DROP ROLE ON *.* TO `icadmin`@`%` WITH GRANT OPTION                                                                                                                                                                                                                                                                                                                                                                 |
| GRANT APPLICATION_PASSWORD_ADMIN,AUDIT_ABORT_EXEMPT,AUDIT_ADMIN,AUTHENTICATION_POLICY_ADMIN,BACKUP_ADMIN,BINLOG_ADMIN,BINLOG_ENCRYPTION_ADMIN,CLONE_ADMIN,CONNECTION_ADMIN,ENCRYPTION_KEY_ADMIN,FIREWALL_EXEMPT,FLUSH_OPTIMIZER_COSTS,FLUSH_STATUS,FLUSH_TABLES,FLUSH_USER_RESOURCES,GROUP_REPLICATION_ADMIN,GROUP_REPLICATION_STREAM,INNODB_REDO_LOG_ARCHIVE,INNODB_REDO_LOG_ENABLE,PASSWORDLESS_USER_ADMIN,PERSIST_RO_VARIABLES_ADMIN,REPLICATION_APPLIER,REPLICATION_SLAVE_ADMIN,RESOURCE_GROUP_ADMIN,RESOURCE_GROUP_USER,ROLE_ADMIN,SENSITIVE_VARIABLES_OBSERVER,SERVICE_CONNECTION_ADMIN,SESSION_VARIABLES_ADMIN,SET_USER_ID,SHOW_ROUTINE,SYSTEM_USER,SYSTEM_VARIABLES_ADMIN,TABLE_ENCRYPTION_ADMIN,XA_RECOVER_ADMIN ON *.* TO `icadmin`@`%` WITH GRANT OPTION |
| GRANT INSERT, UPDATE, DELETE ON `mysql`.* TO `icadmin`@`%` WITH GRANT OPTION                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        |
| GRANT INSERT, UPDATE, DELETE, CREATE, DROP, REFERENCES, INDEX, ALTER, CREATE TEMPORARY TABLES, LOCK TABLES, EXECUTE, CREATE VIEW, SHOW VIEW, CREATE ROUTINE, ALTER ROUTINE, EVENT, TRIGGER ON `mysql_innodb_cluster_metadata`.* TO `icadmin`@`%` WITH GRANT OPTION                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  |
| GRANT INSERT, UPDATE, DELETE, CREATE, DROP, REFERENCES, INDEX, ALTER, CREATE TEMPORARY TABLES, LOCK TABLES, EXECUTE, CREATE VIEW, SHOW VIEW, CREATE ROUTINE, ALTER ROUTINE, EVENT, TRIGGER ON `mysql_innodb_cluster_metadata_bkp`.* TO `icadmin`@`%` WITH GRANT OPTION                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              |
| GRANT INSERT, UPDATE, DELETE, CREATE, DROP, REFERENCES, INDEX, ALTER, CREATE TEMPORARY TABLES, LOCK TABLES, EXECUTE, CREATE VIEW, SHOW VIEW, CREATE ROUTINE, ALTER ROUTINE, EVENT, TRIGGER ON `mysql_innodb_cluster_metadata_previous`.* TO `icadmin`@`%` WITH GRANT OPTION                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         |
+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
```

What privileges could possibly be missing?

> I have tried it both ways - with and without the clusterAdmin name (`icadmin`), but it still fails with the same issue. 

OK.

> The privileges on the replicaCluster primary are the same. We don't allow remote connectivity with root, but I have logged into the host with the `icadmin` account and rerun dba.configureInstance(), and there are no issues. Additionally, if there were issues with the instance configuration, they would/should have been raised during the initial check (as happened when I completely wiped that instance and let mysqlsh reclone it, multiple times).

Correct.

> The privileges were initially granted with the same `ClusterSet.setupAdminAccount()` command, and even after a rerun across the current ClusterSet as well as the node that I intend to add, they are the same:

That command will create an account on the ClusterSet members, not on the instance you want to use to create a new Replica Cluster. How did you create the 'icadmin' account at the target instance? Manually, I suppose? Can you please confirm that?

> What privileges could possibly be missing?

That list of privileges is the list for the 'icadmin' account at the instance you want to use to create a new Replica Cluster? i.e. the one used when you run  `clsset.createReplicaCluster("icadmin@REPLICA_CLUSTER_PRIMARY_HOST:PORT", "cluster2")` ?

I did the following test:

  - Deploy 2 plain MySQL servers: server1, server2
  - Connect to server1 and create an admin account: dba.configureInstance("root@server1:3306", {clusterAdmin: "test"})
  - Connect using the "test" account to server1 and create a Cluster and ClusterSet
  - Create an account named "test" on server2, and grant it the exact same list of privileges you have shared.
  - Create a Replica Cluster: clusterset.createReplicaCluster("server2:3306", "replica_cluster")

And it works as expected. It didn't hit the issue you've reported.