Bug #108472 addInstance should have an option to block/retry on too many clone operations
Submitted: 13 Sep 2022 12:21 Modified: 13 Sep 2022 12:32
Reporter: Jay Janssen Email Updates:
Status: Verified Impact on me:
None 
Category:Shell AdminAPI InnoDB Cluster / ReplicaSet Severity:S4 (Feature request)
Version:8.0.30 OS:Any
Assigned to: CPU Architecture:Any

[13 Sep 2022 12:21] Jay Janssen
Description:
When joining more than one instance simultaneously to a cluster using addInstance (via automation), I get this error if a clone is already running:

* Configuring ClusterSet managed replication channel...
** Changing replication source of 10.170.254.92:3306 to 10.162.254.74:3306

Monitoring recovery process of the new cluster member. Press ^C to stop monitoring and let it continue in background.
ESC[1mClone based state recovery is now in progress.ESC[0m

ESC[36mNOTE: ESC[0mA server restart is expected to happen as part of the clone process. If the
server does not support the RESTART command or does not come back after a
while, you may need to manually start it back.

* Waiting for clone to finish...
ESC[36mNOTE: ESC[0m10.170.254.92:3306 is being cloned from 10.170.255.16:3306
ESC[31mERROR: ESC[0mThe clone process has failed: Clone Donor Error: 3634 : Too many concurrent clone operations. Maximum allowed - 1.. (3862)
ESC[31mERROR: ESC[0mClone Donor Error: 3634 : Too many concurrent clone operations. Maximum allowed - 1..

It'd be helpful if addInstance had an option to wait for cloning to become available rather than just exiting.  

How to repeat:
Start two instances at the same time on a cluster that needs a non-trivial amount of time to clone.  

Suggested fix:
add a 'waitClone' option that takes a maximum number of seconds to wait for cloning to become available.  

cluster.addInstance('new instance', {waitClone: 86400})
[13 Sep 2022 12:32] MySQL Verification Team
Hello Jay Janssen,

Thank you for the feature request!

regards,
Umesh