MySQL Bugs: #103206: Further possible improvements to native cloning

Bug #103206	Further possible improvements to native cloning
Submitted:	5 Apr 2021 12:30	Modified:	2 Oct 2021 10:37
Reporter:	Simon Mudd (OCA)	Email Updates:
Status:	Verified	Impact on me:	None
Category:	MySQL Server: Clone Plugin	Severity:	S3 (Non-critical)
Version:	8.0.23	OS:	Any
Assigned to:		CPU Architecture:	Any
Tags:	allow from multiple sources, keep trying

Description:
MySQL native cloning is the new recommended way to clone a server from existing running systems. However it still has some limitations. Others have been reported before but I'm not sure these suggestions have been.

Typically you follow the steps in https://dev.mysql.com/doc/refman/8.0/en/clone-plugin-remote.html and everything works fine.

This assumes an error free single clone is required and everything will work.

In practice there are several wrong assumptions here:
* a single clone is needed: multiple replicas may be needed to replace existing ones or to expand/extend the cluster
* the CLONE INSTANCE command will always succeed. For a variety of reasons it may not, one being getting an error like:

root@recipient [(none)]> clone instance from 'user'@'donor':3306 identified by 'somesecret' require ssl;
ERROR 3862 (HY000): Clone Donor Error: 3634 : Too many concurrent clone operations. Maximum allowed - 1..

The recipient server has to be configured with a clone_valid_donor_list. This seems surprising given I have to specify from which donor I want to clone anyway. I think this is intended as a security enhancement, but I am not convinced that it is ideal.

However, If we have this donor_list why not use it and attempt to clone from any of the servers in that list?

How to repeat:
Look at docs and try cloning new servers. Think past the basic case of doing a single clone on its own.

Automation hides many of these issues but when they crop up they are still quite frustrating and if you need to handle cloning manually you realise life would be easier if they were resolved.

Suggested fix:
Consider making the CLONE INSTANCE command not require a host:port, but use the values provided in clone_valid_donor_list.

Consider extending the CLONE INSTANCE command so that multiple instances can be referenced.
e.g. clone instance from 'clone_user'@'host1':3306, 'host2':3306, ..., 'hostn':3306 identified by ...;

It is unlikely a setup will use different usernames if trying to clone from different instances: grants should be identical.

Consider allowing the CLONE INSTANCE command to retry (like replicas keep trying to connect to their masters). Having some sort of exponential backoff might be good so have some extra settings such as:

clone_initial_retry_seconds (30)
clone_maximum_retry_seconds (600)
clone_backoff_factor (1.5) applied after trying all instances referenced in clone_valid_donor_list.
clone_maximum_failed_attempts (1000) # perhaps

If you do this you may need some status metrics: clone_attempt_number or clone_failed_attempts which the user can monitor.

Finally consider allowing more than one simultaneous clone to take place from the same donor and provide a clone_maximum_concurrent_recipients (1) setting. A modern server may have sufficient i/o, cpu or network bandwidth to handle cloning to multiple recipients at once and the user would like to take advantage of those resources if possible.

Hello Simon,

Thank you for the enhancement request!

regards,
Umesh

Adding some more things to consider:

1. set global clone_enable_compression = 1; DOES NOT appear to work with an existing session. That is the existing connections don't change.
* improve documentation to explain when this setting will be used
* consider reading the setting for each "request" and if the setting changes drop the connection and reconnect if this is a network setting triggered at connection time only, and then continuing normally. If this does not require reconnecting then make the thread check the setting after each operation and adjust the behaviour appropriately

2. other configuration settings like clone-max-data-bandwidth and clone-max-network-bandwidth also appear to adjust behaviour but the numbers I saw when throttling the copy did not seem to adjust very closely to the selected values, with data_speed and network_speed shown in performance_schema.clone_progress staying above the configured values. Again the reason for this is not clear so improved documentation would be helpful and if necessary check the settings more frequently to allow for more dynamic configuration.

3. Throttling output on the source.

I have seen that the source server may get overloaded. Is this is a source of replication (master or intermediate master) the primary concern of an administrator is that "replication" works allowing other downstream replicas to keep up with the changes on the master. There appears to be no way to throttle writes on the source irrespective of what the recipient wants to do. That's not ideal.

I can think of a setting which checks replication delay and tries to ensure that the replication delay is below a threshold, throttling back on cloning during this time, or simply the similar settings which appear to be configurable on the recipient to control network or i/o traffic, they should be also be optionally configurable on the source to allow throttling of downstream cloning rates.

4. unexplained variable network traffic speeds

I see moments when the FILE COPY phase network and data speeds drop significantly. It's not clear why this is. The variation can change from 400 MB/s to 4 MB/s on 10 Gbe networking. I assume that this may be due to issues on the source but there's no visibility on why the behaviour changes so significantly.

5. clone_progress does not change on the source server. The copy I see there is from the clone that server made from another server. It would be useful perhaps to provide a different table for the source server to show it's view of what is going on.

6. Consider adding source host/port and version to performance_schema.clone_progress. This provides a simple additional bit of information showing where the clone is being made from. Given this table is not modified once the clone operation has completed it provides a record of not only when the clone took place but from which server and can be useful.

7. clone_progress has no detailed indication of progress. It would be good to have an extra column COMPLETED_STEPS / JOBS / TASKS etc which indicates a completed step in the process and updates this as needed.  If the number of steps is described for each stage then it's a good way to see how progress is completing.   That can be inferred from the  estimate / data/ network (at least in the FILE COPY stage but that sort of assumes all files are the same which is clearly not the case.