Description:
Table of Contents
=================
1 Connecters inserting large amount of data before the TTL expires, after a resharding metadata update, can cause inconsistency.
1.1 Credits
1.2 Problem Description
1.3 Solution
1.4 Implementation
1 Connecters inserting large amount of data before the TTL expires, after a resharding metadata update, can cause inconsistency.
=================================================================================================================================
1.1 Credits
------------
Jack Tom in the community raised this issue.
Mats gave inputs on possible angles and solution to the problem.
1.2 Problem Description
------------------------
After a shard move or a split happens, the connectors update the new sharding
metadata information post the expiry of the TTL. But during this period before
TTL expiry a connector can easily insert large amount of data. This data would
not have been bound by the actions that take place during the resharding. This
can result in inconsistency between the sharding definition and the data on the
shard.
1.3 Solution
-------------
A possible solution would be to block all the connectors that are connecting to
the old shard after the resharding until the TTL expires. Block all the connectors,
wait until the TTL expires, update the metadata and unblock all the connectors.
1.4 Implementation
-------------------
Since there is no lock server we will have to lock all the sharded tables individually.
How to repeat:
Perform a shard split or a shard move in Fabric with continuous heavy load on the connector. In the cases where the split / move finishes before TTL expiry the connector continues to insert data into the old shard until TTL expiry (even after the metadata has switched).
Description: Table of Contents ================= 1 Connecters inserting large amount of data before the TTL expires, after a resharding metadata update, can cause inconsistency. 1.1 Credits 1.2 Problem Description 1.3 Solution 1.4 Implementation 1 Connecters inserting large amount of data before the TTL expires, after a resharding metadata update, can cause inconsistency. ================================================================================================================================= 1.1 Credits ------------ Jack Tom in the community raised this issue. Mats gave inputs on possible angles and solution to the problem. 1.2 Problem Description ------------------------ After a shard move or a split happens, the connectors update the new sharding metadata information post the expiry of the TTL. But during this period before TTL expiry a connector can easily insert large amount of data. This data would not have been bound by the actions that take place during the resharding. This can result in inconsistency between the sharding definition and the data on the shard. 1.3 Solution ------------- A possible solution would be to block all the connectors that are connecting to the old shard after the resharding until the TTL expires. Block all the connectors, wait until the TTL expires, update the metadata and unblock all the connectors. 1.4 Implementation ------------------- Since there is no lock server we will have to lock all the sharded tables individually. How to repeat: Perform a shard split or a shard move in Fabric with continuous heavy load on the connector. In the cases where the split / move finishes before TTL expiry the connector continues to insert data into the old shard until TTL expiry (even after the metadata has switched).