Description:
native cloning is really good and really fast. I often see 800-900 MB/s on 10gbe networking between servers. setup and configuration is described here https://dev.mysql.com/doc/refman/8.4/en/clone-plugin.html (for 8.4 but 8.0 is the same).
I noticed that a clone I kicked off was running slowly:
2024-11-07T17:40:43.630933Z 7 [Warning] [MY-013460] [InnoDB] Clone removing all user data for provisioning: Finished
2024-11-07T17:40:43.632671Z 7 [Note] [MY-013272] [Clone] Plugin Clone reported: 'Client: Command COM_INIT.'
2024-11-07T17:40:44.132973Z 7 [Note] [MY-013458] [InnoDB] Clone Apply State Change : Number of tasks = 1
2024-11-07T17:40:44.133222Z 7 [Note] [MY-013458] [InnoDB] Clone Apply State FILE COPY:
2024-11-07T17:40:44.133332Z 7 [Note] [MY-011978] [InnoDB] Clone estimated size: 7.64 TiB Available space: 9.90 TiB
2024-11-07T17:41:53.091422Z 7 [Note] [MY-013272] [Clone] Plugin Clone reported: 'Client: Tune Threads from: 1 to: 2 prev: 1 target: 2.'
2024-11-07T17:41:53.187854Z 0 [Note] [MY-013272] [Clone] Plugin Clone reported: 'Client: Task Connect.'
2024-11-07T17:41:53.193740Z 0 [Note] [MY-013457] [InnoDB] Clone Apply Begin Task ID: 1
2024-11-07T17:41:53.193809Z 0 [Note] [MY-013272] [Clone] Plugin Clone reported: 'Client: Command COM_ATTACH.'
2024-11-07T17:41:58.119142Z 7 [Note] [MY-013272] [Clone] Plugin Clone reported: 'Client: Tune stop, Data: 201 MiB/sec, Target: 226 MiB/sec..'
I was expecting something much higher. I then checked with a colleague:
> Yes for 1010. There is QoS for Read. The whole 200MB consumed.
That is this system, both source and destination instances were connected to a (different) storage appliance.
I asked for the throttling limit to be removed, but see that the logging shows "Tune stop"
and the documentation says that once the max speed has been achieved that no further changes are made.
This is a shame as I had the limit removed and in theory could have obtained a 3-4 x increase in performance.
How to repeat:
- Do native cloning where storage i/o is throttled.
- What the the mysqld logging. See that it stabalises and does not try to modify settings thereafter.
Suggested fix:
Make this a little more dynamic.
- If there seems to be a sufficient improvement in throughput performance despite having stopped previously consider trying to increase the target value again. Having a few extra threads running when maybe they are not all fully active does not rally matter as in the end all the server is doing is using them to copy data. Wasting resources by not using these threads seem exactly that to be a waste when they could be used (if indicated by an increase in performance.)
- it might also be useful to log periodically (e.g. every 5 minutes) the current throughput rate being obtained as once tuning stops no i/o or network bandwidth values are logged to the log file.
[ I notice that the system does not reduce the number of threads used. That may not be necessary but if the number of threads is high that might be worth considering. ]
