MySQL Bugs: #47674: MySQL Cluster : Manually applying binlog zaps unmodified columns

Bug #47674	MySQL Cluster : Manually applying binlog zaps unmodified columns
Submitted:	28 Sep 2009 11:37	Modified:	8 Oct 2009 17:23
Reporter:	Frazer Clement	Email Updates:
Status:	Closed	Impact on me:	None
Category:	MySQL Cluster: Replication	Severity:	S2 (Serious)
Version:	mysql-5.1-telco-6.2	OS:	Any
Assigned to:	Frazer Clement	CPU Architecture:	Any

Description:
MySQL Cluster supports 2 options to modify how Binlogs are recorded :

(1) ndb-log-update-as-write
(2) ndb-log-updated-only

By default both are on.

Option (1) logs row update operations as WRITE_ROW events containing only after-image data.  This reduces the Binlog size generated.
Option (2) logs only the primary key and modified columns when logging an update.  This reduces the Binlog size generated.

When applying WRITE_ROW events from a Binlog, it is important that the applier checks whether the row exists or not.
a) If the row does not exist :
  Applier treats WRITE_ROW as an insert, setting values for all columns.
  (All columns should be present in event if it was an INSERT at the source cluster).

b) If the row already exists :
  Applier treats WRITE_ROW as an update, *only* setting values for columns present in the WRITE_ROW event.

It seems that in the b) scenario, where the row already exists, manually applying Binlog with columns not in the WRITE_ROW event (due to options (1) and (2)), results in the 'missing' columns being trampled to their default values (e.g. NULL, 0 or whatever).

How to repeat:
# On Mysqld with options (1) and (2) on (default)

# Create Binlog with UPDATE not containing all columns in existing row
mysql> RESET MASTER;
mysql> CREATE TABLE ba (ks int primary key, st int, lp int) engine = ndb;
mysql> INSERT INTO ba VALUES (1,1,1);
mysql> INSERT INTO ba VALUES (2,2,2);
mysql> UPDATE ba SET st=20 WHERE ks=2;
mysql> FLUSH LOGS;

mysql> DROP TABLE ba;

# Manually apply Binlog
shell> mysqlbinlog mysqld/master-bin.000001 | mysql -uroot

# Inspect table contents
mysql> SELECT * from ba ORDER BY ks;

# Table will have trampled value for lp, e.g. NULL

Suggested fix:
Replication slave has 'special case' for this scenario, causing REPLACE command to behave differently when applying Binlog in slave.

Extend this 'special case' to the manually-applied Binlog scenario.

Proposed patch on top of fix to bug#47323

Attachment: bug47674.patch (text/x-patch), 7.34 KiB.

Notes :

The testcase is testing the case where an update is logged as a write, and not all columns in the table are updated.  Before this patch, the unaffected column (st) is trampled to NULL.

isManualBinlogExec() implementation is quite nasty, but I'm not sure of any other way to determine whether we're executing BINLOG via the mysql client.  Ideally there would be something to check whether we were executing a RBR BINLOG statement.

In general, I think that most of the checks on thd->slave should apply to the manually-applied Binlog so that it's possible for users to get the same effect when applying Binlog manually.

These checks currently (in 6.2 source) cover :
 - Source ServerId used when logging the update
   Currently, manually applied Binlog will be binlogged with the executing server as the source server.
 - HA_EXTRA_WRITE_CAN_REPLACE activation with unique indexes on table
   Currently, manually applied Binlog will hit duplicate key errors when the table to be written to has a unique index (bug#46662)
 - Logging and generation of ndb_apply_status updates

In later releases, they also include code for conflict detection and resolution.

So other bug reports may also make use of the new isManualBinlogExec() function.

Updated patch with embedded server workaround

Attachment: bug47674.patch (text/x-patch), 7.59 KiB.

A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/86005

3018 Frazer Clement	2009-10-07
      Bug#47674 : MySQL Cluster : Manually applying binlog zaps unmodified columns
      modified:
        mysql-test/suite/ndb_binlog/r/ndb_binlog_variants.result
        mysql-test/suite/ndb_binlog/t/ndb_binlog_variants.test
        sql/ha_ndbcluster.cc
        sql/ha_ndbcluster.h

A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/86044

3019 Frazer Clement	2009-10-07
      Bug#47674 : MySQL Cluster : Manually applying binlog zaps unmodified columns
      modified:
        mysql-test/suite/ndb_binlog/r/ndb_binlog_variants.result
        mysql-test/suite/ndb_binlog/t/ndb_binlog_variants.test
        sql/ha_ndbcluster.cc
        sql/ha_ndbcluster.h

Fix pushed to 
6.2.19
6.3.28
7.0.9
7.1.0

Documented bugfix in the NDB-6.2.19, 6.3.28, and 7.0.9 changelogs, as follows:

        When recording a binary log using the --ndb-log-update-as-write
        and --ndb-log-updated-only options (both enabled by default) and
        later attempting to apply that binary log with mysqlbinlog, any
        operations played back from the log that updated only some (but
        not all) columns caused any columns that were not updated to be
        reset to their default values.

Closed.

A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/87008

3018 Martin Skold	2009-10-15 [merge]
      Merge
      modified:
        config/ac-macros/ha_ndbcluster.m4
        mysql-test/suite/ndb/my.cnf
        mysql-test/suite/ndb/r/ndb_config.result
        mysql-test/suite/ndb_binlog/r/ndb_binlog_variants.result
        mysql-test/suite/ndb_binlog/t/ndb_binlog_variants.test
        mysql-test/suite/ndb_team/r/ndb_dd_backuprestore.result
        sql/ha_ndbcluster.cc
        sql/ha_ndbcluster.h
        storage/ndb/include/kernel/signaldata/ContinueFragmented.hpp
        storage/ndb/include/kernel/signaldata/DumpStateOrd.hpp
        storage/ndb/include/kernel/signaldata/NodeFailRep.hpp
        storage/ndb/include/ndb_global.h.in
        storage/ndb/include/ndbapi/NdbDictionary.hpp
        storage/ndb/include/ndbapi/NdbOperation.hpp
        storage/ndb/ndbapi-examples/ndbapi_scan/ndbapi_scan.cpp
        storage/ndb/src/kernel/blocks/backup/Backup.cpp
        storage/ndb/src/kernel/blocks/backup/Backup.hpp
        storage/ndb/src/kernel/blocks/cmvmi/Cmvmi.cpp
        storage/ndb/src/kernel/blocks/cmvmi/Cmvmi.hpp
        storage/ndb/src/kernel/blocks/dbdict/Dbdict.cpp
        storage/ndb/src/kernel/blocks/dbdict/Dbdict.hpp
        storage/ndb/src/kernel/blocks/dbdict/printSchemaFile.cpp
        storage/ndb/src/kernel/blocks/dbdih/printSysfile.cpp
        storage/ndb/src/kernel/blocks/dbdih/printSysfile/printSysfile.cpp
        storage/ndb/src/kernel/blocks/dblqh/DblqhMain.cpp
        storage/ndb/src/kernel/blocks/dblqh/redoLogReader/reader.cpp
        storage/ndb/src/kernel/blocks/dbtc/Dbtc.hpp
        storage/ndb/src/kernel/blocks/dbtc/DbtcMain.cpp
        storage/ndb/src/kernel/blocks/dbtup/Dbtup.hpp
        storage/ndb/src/kernel/blocks/dbtup/DbtupGen.cpp
        storage/ndb/src/kernel/blocks/dbtup/tuppage.hpp
        storage/ndb/src/kernel/blocks/dbtux/Dbtux.hpp
        storage/ndb/src/kernel/blocks/dbutil/DbUtil.cpp
        storage/ndb/src/kernel/blocks/dbutil/DbUtil.hpp
        storage/ndb/src/kernel/blocks/lgman.cpp
        storage/ndb/src/kernel/blocks/lgman.hpp
        storage/ndb/src/kernel/blocks/ndbcntr/NdbcntrMain.cpp
        storage/ndb/src/kernel/blocks/qmgr/QmgrMain.cpp
        storage/ndb/src/kernel/blocks/suma/Suma.cpp
        storage/ndb/src/kernel/blocks/suma/Suma.hpp
        storage/ndb/src/kernel/blocks/trix/Trix.hpp
        storage/ndb/src/kernel/blocks/tsman.cpp
        storage/ndb/src/kernel/blocks/tsman.hpp
        storage/ndb/src/kernel/vm/DLFifoList.hpp
        storage/ndb/src/kernel/vm/DLHashTable.hpp
        storage/ndb/src/kernel/vm/DLList.hpp
        storage/ndb/src/kernel/vm/DataBuffer.hpp
        storage/ndb/src/kernel/vm/SimulatedBlock.cpp
        storage/ndb/src/kernel/vm/SimulatedBlock.hpp
        storage/ndb/src/mgmapi/LocalConfig.cpp
        storage/ndb/src/mgmapi/Makefile.am
        storage/ndb/src/mgmsrv/ConfigInfo.cpp
        storage/ndb/src/ndbapi/NdbDictionary.cpp
        storage/ndb/src/ndbapi/NdbOperation.cpp
        storage/ndb/test/ndbapi/testNdbApi.cpp
        storage/ndb/test/run-test/daily-basic-tests.txt