Bug #73546 Resharding can cause inconsistency during high loads
| Submitted: | 12 Aug 2014 8:36 | Modified: | 5 Sep 2014 5:08 |
|---|---|---|---|
| Reporter: | V Venkateswaran | Email Updates: | |
| Status: | Closed | Impact on me: | |
| Category: | MySQL Fabric | Severity: | S2 (Serious) |
| Version: | 1.4 | OS: | Any |
| Assigned to: | CPU Architecture: | Any | |
| Tags: | move, sharding, split | ||
[12 Aug 2014 9:15]
Mats Kindahl
Thank you for the bug report. Verified as described.
[5 Sep 2014 5:08]
Philip Olson
Fixed as of the upcoming MySQL Utilities 1.5.2 release, and here's the changelog entry: Connectors inserting large amount of data before the TTL expires, after a resharding metadata update, could cause inconsistencies between the sharding definition and the data on the shard. Thank you for the bug report.
Portions of this website are copyright © 2001, 2002 The PHP Group
Page generated in 0.041 sec. using MySQL 8.4.7-u6-cloud
Timestamp references displayed by the system are UTC.
Content reproduced on this site is the property of the
respective copyright holders.
It is not reviewed in advance
by Oracle and does not necessarily represent the opinion of
Oracle or any other party.
Description: Table of Contents ================= 1 Connecters inserting large amount of data before the TTL expires, after a resharding metadata update, can cause inconsistency. 1.1 Credits 1.2 Problem Description 1.3 Solution 1.4 Implementation 1 Connecters inserting large amount of data before the TTL expires, after a resharding metadata update, can cause inconsistency. ================================================================================================================================= 1.1 Credits ------------ Jack Tom in the community raised this issue. Mats gave inputs on possible angles and solution to the problem. 1.2 Problem Description ------------------------ After a shard move or a split happens, the connectors update the new sharding metadata information post the expiry of the TTL. But during this period before TTL expiry a connector can easily insert large amount of data. This data would not have been bound by the actions that take place during the resharding. This can result in inconsistency between the sharding definition and the data on the shard. 1.3 Solution ------------- A possible solution would be to block all the connectors that are connecting to the old shard after the resharding until the TTL expires. Block all the connectors, wait until the TTL expires, update the metadata and unblock all the connectors. 1.4 Implementation ------------------- Since there is no lock server we will have to lock all the sharded tables individually. How to repeat: Perform a shard split or a shard move in Fabric with continuous heavy load on the connector. In the cases where the split / move finishes before TTL expiry the connector continues to insert data into the old shard until TTL expiry (even after the metadata has switched).