Description:
With binlog_row_value_options=PARTIAL_JSON, the source can log a JSON diff
REPLACE whose value equals the value already at that path. When the replica
applies the earlier diffs of the same row in place (binary update), it rejects
this diff with ER_COULD_NOT_APPLY_JSON_DIFF (3648). The replica's applier
stops. Replaying the binary log with mysqlbinlog | mysql stops at the same
event.
The logged diff is valid: applied to the before-image it gives the source's
row. The DOM applier would accept it; only the in-place path rejects it.
Three conditions are needed (each was checked by changing one of them):
- a JSON_SET that auto-wraps a scalar ('$.arr[5]' on a string member), so the
source falls back to the DOM and logs REPLACE '$.arr' <- ["xxxxx", "v"];
- a later path in the same JSON_SET that writes the value already there
('$.arr[1]', 'v'); with 'w' instead, the row replicates correctly;
- an earlier in-place update that left free space after "arr" in the stored
document (here "nested" is replaced by null), so the replica can apply the
first diff in place.
Reproduced on trunk 3b99be40f8a (built from source, RelWithDebInfo) and on
26.7.0. The default binlog_row_image=FULL was used.
How to repeat:
Start two servers (source and replica), both with:
--log-bin=binlog --gtid-mode=ON --enforce-gtid-consistency=ON
--binlog-row-value-options=PARTIAL_JSON
and --server-id=1 on the source, --server-id=2 on the replica,
and on the replica:
CHANGE REPLICATION SOURCE TO SOURCE_HOST='127.0.0.1', SOURCE_PORT=<source port>,
SOURCE_USER='root', SOURCE_PASSWORD='', SOURCE_AUTO_POSITION=1, GET_SOURCE_PUBLIC_KEY=1;
START REPLICA;
On the source:
FLUSH BINARY LOGS;
CREATE DATABASE d;
CREATE TABLE d.t (id INT PRIMARY KEY, doc JSON);
INSERT INTO d.t VALUES (1, '{"nested": {"x": 1}, "arr": "xxxxx"}');
UPDATE d.t SET doc = JSON_REPLACE(doc, '$.nested', CAST('null' AS JSON)) WHERE id = 1;
UPDATE d.t SET doc = JSON_SET(JSON_SET(doc, '$.arr[5]', 'v'), '$.arr[1]', 'v') WHERE id = 1;
INSERT INTO d.t VALUES (2, '{"after": true}');
FLUSH BINARY LOGS;
Then compare:
SELECT id, doc FROM d.t ORDER BY id; -- on source and replica
SELECT LAST_ERROR_NUMBER, LAST_ERROR_MESSAGE
FROM performance_schema.replication_applier_status_by_worker
WHERE LAST_ERROR_NUMBER <> 0; -- on the replica
Observed on trunk (26.10.0):
source : 1 {"arr": ["xxxxx", "v"], "nested": null}
2 {"after": true}
replica: 1 {"arr": "xxxxx", "nested": null}
replica error: 3648 Worker 1 failed executing transaction '...:5' at source
log binlog.000002, end_log_pos 1555; Could not execute Update_rows_partial
event on table d.t; Could not apply JSON diff in table t, column doc.,
Error_code: 3648
The second UPDATE is logged as (mysqlbinlog -vv --base64-output=DECODE-ROWS):
### UPDATE `d`.`t`
### WHERE
### @1=1
### @2='{"arr": "xxxxx", "nested": null}'
### SET
### @1=1
### @2=JSON_REPLACE(@2, '$.arr', CAST('["xxxxx", "v"]' AS JSON),
### '$.arr[1]', 'v')
Replaying the source's binlog.000002 into a new server (started without
PARTIAL_JSON) stops at the same event:
$ mysqlbinlog binlog.000002 | mysql -uroot -h127.0.0.1 -P<port>
ERROR 3648 (HY000) at line 130: Could not apply JSON diff in table t, column doc.
restored: 1 {"arr": "xxxxx", "nested": null}
Expected: the replica and the replayed server end with the source's rows.
Control: with '$.arr[1]', 'w' in the second UPDATE, the replica and the replay
both end with 1 {"arr": ["xxxxx", "w"], "nested": null} and 2 {"after": true}.
Suggested fix:
Suggested fix:
Source side: in Item_func_json_set_replace::val_json, the DOM branch logs a
REPLACE diff for every path it finds, also when the value does not change
(sql/item_json_func.cc, lines 2363-2369):
https://github.com/mysql/mysql-server/blob/3b99be40f8a31d9396cad10accfd3ba51647d613/sql/it...
The binary branch logs a REPLACE only when replaced_path is true (lines 2231-2234).
Replica side: apply_json_diffs applies REPLACE in place and treats
replaced_path == false as "path not found":
https://github.com/mysql/mysql-server/blob/3b99be40f8a31d9396cad10accfd3ba51647d613/sql/it...
if (partially_updated) {
if (!replaced_path) return enum_json_diff_status::REJECTED;
replaced_path comes from Json_wrapper::attempt_binary_update, which sets it to
the "changed" flag of Value::update_in_shadow:
https://github.com/mysql/mysql-server/blob/3b99be40f8a31d9396cad10accfd3ba51647d613/sql-co...
update_in_shadow sets "changed" only when the new bytes differ from the old
ones (memcmp):
https://github.com/mysql/mysql-server/blob/3b99be40f8a31d9396cad10accfd3ba51647d613/sql-co...
So a REPLACE with an identical value at an existing path is rejected.
Suggested fix: let attempt_binary_update report "path found" separately from
"bytes changed", and in apply_json_diffs reject only when the path was not
found. Not logging no-op REPLACE diffs on the source would avoid the case for
new binary logs, but binary logs already written still contain such diffs.
Description: With binlog_row_value_options=PARTIAL_JSON, the source can log a JSON diff REPLACE whose value equals the value already at that path. When the replica applies the earlier diffs of the same row in place (binary update), it rejects this diff with ER_COULD_NOT_APPLY_JSON_DIFF (3648). The replica's applier stops. Replaying the binary log with mysqlbinlog | mysql stops at the same event. The logged diff is valid: applied to the before-image it gives the source's row. The DOM applier would accept it; only the in-place path rejects it. Three conditions are needed (each was checked by changing one of them): - a JSON_SET that auto-wraps a scalar ('$.arr[5]' on a string member), so the source falls back to the DOM and logs REPLACE '$.arr' <- ["xxxxx", "v"]; - a later path in the same JSON_SET that writes the value already there ('$.arr[1]', 'v'); with 'w' instead, the row replicates correctly; - an earlier in-place update that left free space after "arr" in the stored document (here "nested" is replaced by null), so the replica can apply the first diff in place. Reproduced on trunk 3b99be40f8a (built from source, RelWithDebInfo) and on 26.7.0. The default binlog_row_image=FULL was used. How to repeat: Start two servers (source and replica), both with: --log-bin=binlog --gtid-mode=ON --enforce-gtid-consistency=ON --binlog-row-value-options=PARTIAL_JSON and --server-id=1 on the source, --server-id=2 on the replica, and on the replica: CHANGE REPLICATION SOURCE TO SOURCE_HOST='127.0.0.1', SOURCE_PORT=<source port>, SOURCE_USER='root', SOURCE_PASSWORD='', SOURCE_AUTO_POSITION=1, GET_SOURCE_PUBLIC_KEY=1; START REPLICA; On the source: FLUSH BINARY LOGS; CREATE DATABASE d; CREATE TABLE d.t (id INT PRIMARY KEY, doc JSON); INSERT INTO d.t VALUES (1, '{"nested": {"x": 1}, "arr": "xxxxx"}'); UPDATE d.t SET doc = JSON_REPLACE(doc, '$.nested', CAST('null' AS JSON)) WHERE id = 1; UPDATE d.t SET doc = JSON_SET(JSON_SET(doc, '$.arr[5]', 'v'), '$.arr[1]', 'v') WHERE id = 1; INSERT INTO d.t VALUES (2, '{"after": true}'); FLUSH BINARY LOGS; Then compare: SELECT id, doc FROM d.t ORDER BY id; -- on source and replica SELECT LAST_ERROR_NUMBER, LAST_ERROR_MESSAGE FROM performance_schema.replication_applier_status_by_worker WHERE LAST_ERROR_NUMBER <> 0; -- on the replica Observed on trunk (26.10.0): source : 1 {"arr": ["xxxxx", "v"], "nested": null} 2 {"after": true} replica: 1 {"arr": "xxxxx", "nested": null} replica error: 3648 Worker 1 failed executing transaction '...:5' at source log binlog.000002, end_log_pos 1555; Could not execute Update_rows_partial event on table d.t; Could not apply JSON diff in table t, column doc., Error_code: 3648 The second UPDATE is logged as (mysqlbinlog -vv --base64-output=DECODE-ROWS): ### UPDATE `d`.`t` ### WHERE ### @1=1 ### @2='{"arr": "xxxxx", "nested": null}' ### SET ### @1=1 ### @2=JSON_REPLACE(@2, '$.arr', CAST('["xxxxx", "v"]' AS JSON), ### '$.arr[1]', 'v') Replaying the source's binlog.000002 into a new server (started without PARTIAL_JSON) stops at the same event: $ mysqlbinlog binlog.000002 | mysql -uroot -h127.0.0.1 -P<port> ERROR 3648 (HY000) at line 130: Could not apply JSON diff in table t, column doc. restored: 1 {"arr": "xxxxx", "nested": null} Expected: the replica and the replayed server end with the source's rows. Control: with '$.arr[1]', 'w' in the second UPDATE, the replica and the replay both end with 1 {"arr": ["xxxxx", "w"], "nested": null} and 2 {"after": true}. Suggested fix: Suggested fix: Source side: in Item_func_json_set_replace::val_json, the DOM branch logs a REPLACE diff for every path it finds, also when the value does not change (sql/item_json_func.cc, lines 2363-2369): https://github.com/mysql/mysql-server/blob/3b99be40f8a31d9396cad10accfd3ba51647d613/sql/it... The binary branch logs a REPLACE only when replaced_path is true (lines 2231-2234). Replica side: apply_json_diffs applies REPLACE in place and treats replaced_path == false as "path not found": https://github.com/mysql/mysql-server/blob/3b99be40f8a31d9396cad10accfd3ba51647d613/sql/it... if (partially_updated) { if (!replaced_path) return enum_json_diff_status::REJECTED; replaced_path comes from Json_wrapper::attempt_binary_update, which sets it to the "changed" flag of Value::update_in_shadow: https://github.com/mysql/mysql-server/blob/3b99be40f8a31d9396cad10accfd3ba51647d613/sql-co... update_in_shadow sets "changed" only when the new bytes differ from the old ones (memcmp): https://github.com/mysql/mysql-server/blob/3b99be40f8a31d9396cad10accfd3ba51647d613/sql-co... So a REPLACE with an identical value at an existing path is rejected. Suggested fix: let attempt_binary_update report "path found" separately from "bytes changed", and in apply_json_diffs reject only when the path was not found. Not logging no-op REPLACE diffs on the source would avoid the case for new binary logs, but binary logs already written still contain such diffs.