Bug #121555 PARTIAL_JSON: replica rejects a JSON diff that replaces a value with the same value (error 3648)
Submitted: 20 Oct 3:29
Reporter: Ke Han Email Updates:
Status: Open Impact on me:
None 
Category:MySQL Server: Replication Severity:S2 (Serious)
Version:26.7.0, trunk 3b99be40f8a (26.10.0) OS:Any
Assigned to: CPU Architecture:Any

[20 Oct 3:29] Ke Han
Description:
With binlog_row_value_options=PARTIAL_JSON, the source can log a JSON diff
REPLACE whose value equals the value already at that path. When the replica
applies the earlier diffs of the same row in place (binary update), it rejects
this diff with ER_COULD_NOT_APPLY_JSON_DIFF (3648). The replica's applier
stops. Replaying the binary log with mysqlbinlog | mysql stops at the same
event.

The logged diff is valid: applied to the before-image it gives the source's
row. The DOM applier would accept it; only the in-place path rejects it.

Three conditions are needed (each was checked by changing one of them):
- a JSON_SET that auto-wraps a scalar ('$.arr[5]' on a string member), so the
  source falls back to the DOM and logs REPLACE '$.arr' <- ["xxxxx", "v"];
- a later path in the same JSON_SET that writes the value already there
  ('$.arr[1]', 'v'); with 'w' instead, the row replicates correctly;
- an earlier in-place update that left free space after "arr" in the stored
  document (here "nested" is replaced by null), so the replica can apply the
  first diff in place.

Reproduced on trunk 3b99be40f8a (built from source, RelWithDebInfo) and on
26.7.0. The default binlog_row_image=FULL was used.

How to repeat:
Start two servers (source and replica), both with:
  --log-bin=binlog --gtid-mode=ON --enforce-gtid-consistency=ON
  --binlog-row-value-options=PARTIAL_JSON
and --server-id=1 on the source, --server-id=2 on the replica,
and on the replica:
  CHANGE REPLICATION SOURCE TO SOURCE_HOST='127.0.0.1', SOURCE_PORT=<source port>,
    SOURCE_USER='root', SOURCE_PASSWORD='', SOURCE_AUTO_POSITION=1, GET_SOURCE_PUBLIC_KEY=1;
  START REPLICA;

On the source:
  FLUSH BINARY LOGS;
  CREATE DATABASE d;
  CREATE TABLE d.t (id INT PRIMARY KEY, doc JSON);
  INSERT INTO d.t VALUES (1, '{"nested": {"x": 1}, "arr": "xxxxx"}');
  UPDATE d.t SET doc = JSON_REPLACE(doc, '$.nested', CAST('null' AS JSON)) WHERE id = 1;
  UPDATE d.t SET doc = JSON_SET(JSON_SET(doc, '$.arr[5]', 'v'), '$.arr[1]', 'v') WHERE id = 1;
  INSERT INTO d.t VALUES (2, '{"after": true}');
  FLUSH BINARY LOGS;

Then compare:
  SELECT id, doc FROM d.t ORDER BY id;   -- on source and replica
  SELECT LAST_ERROR_NUMBER, LAST_ERROR_MESSAGE
    FROM performance_schema.replication_applier_status_by_worker
    WHERE LAST_ERROR_NUMBER <> 0;        -- on the replica

Observed on trunk (26.10.0):
  source : 1 {"arr": ["xxxxx", "v"], "nested": null}
           2 {"after": true}
  replica: 1 {"arr": "xxxxx", "nested": null}
  replica error: 3648 Worker 1 failed executing transaction '...:5' at source
    log binlog.000002, end_log_pos 1555; Could not execute Update_rows_partial
    event on table d.t; Could not apply JSON diff in table t, column doc.,
    Error_code: 3648

The second UPDATE is logged as (mysqlbinlog -vv --base64-output=DECODE-ROWS):
  ### UPDATE `d`.`t`
  ### WHERE
  ###   @1=1
  ###   @2='{"arr": "xxxxx", "nested": null}'
  ### SET
  ###   @1=1
  ###   @2=JSON_REPLACE(@2, '$.arr', CAST('["xxxxx", "v"]' AS JSON),
  ###      '$.arr[1]', 'v')

Replaying the source's binlog.000002 into a new server (started without
PARTIAL_JSON) stops at the same event:
  $ mysqlbinlog binlog.000002 | mysql -uroot -h127.0.0.1 -P<port>
  ERROR 3648 (HY000) at line 130: Could not apply JSON diff in table t, column doc.
  restored: 1 {"arr": "xxxxx", "nested": null}

Expected: the replica and the replayed server end with the source's rows.

Control: with '$.arr[1]', 'w' in the second UPDATE, the replica and the replay
both end with 1 {"arr": ["xxxxx", "w"], "nested": null} and 2 {"after": true}.

Suggested fix:
Suggested fix:
Source side: in Item_func_json_set_replace::val_json, the DOM branch logs a
REPLACE diff for every path it finds, also when the value does not change
(sql/item_json_func.cc, lines 2363-2369):
https://github.com/mysql/mysql-server/blob/3b99be40f8a31d9396cad10accfd3ba51647d613/sql/it...
The binary branch logs a REPLACE only when replaced_path is true (lines 2231-2234).

Replica side: apply_json_diffs applies REPLACE in place and treats
replaced_path == false as "path not found":
https://github.com/mysql/mysql-server/blob/3b99be40f8a31d9396cad10accfd3ba51647d613/sql/it...
    if (partially_updated) {
      if (!replaced_path) return enum_json_diff_status::REJECTED;
replaced_path comes from Json_wrapper::attempt_binary_update, which sets it to
the "changed" flag of Value::update_in_shadow:
https://github.com/mysql/mysql-server/blob/3b99be40f8a31d9396cad10accfd3ba51647d613/sql-co...
update_in_shadow sets "changed" only when the new bytes differ from the old
ones (memcmp):
https://github.com/mysql/mysql-server/blob/3b99be40f8a31d9396cad10accfd3ba51647d613/sql-co...
So a REPLACE with an identical value at an existing path is rejected.

Suggested fix: let attempt_binary_update report "path found" separately from
"bytes changed", and in apply_json_diffs reject only when the path was not
found. Not logging no-op REPLACE diffs on the source would avoid the case for
new binary logs, but binary logs already written still contain such diffs.