MySQL Bugs: #52305: Customizing storage of server id inside the anyValue

Bug #52305	Customizing storage of server id inside the anyValue
Submitted:	23 Mar 2010 14:16	Modified:	13 Jul 2010 2:22
Reporter:	Geert Vanderkelen	Email Updates:
Status:	Closed	Impact on me:	None
Category:	MySQL Cluster: Cluster (NDB) storage engine	Severity:	S4 (Feature request)
Version:	mysql-5.1-telco-7.0	OS:	Any
Assigned to:	Frazer Clement	CPU Architecture:	Any

Description:
When binary logging NDB events, MySQL uses OperationOptions::anyValue
member to store the server id. However, other NDB API application could
use the anyValue for anyThing (pun intended).

To help distinguish whether between SQL and API nodes, it would be good
if the server-id is stored in a different way. However, the server id
is a 32-bit integer, and we can't break compatibility.

We have already --ndb-log-orig. This options logs the server id and
epoch the ndb_binlog_index table. It currently doesn't accept values.

Proposal:
Allow a value for --ndb-log-orig which will be the bits to use for
storing the server id. For example, if --ndb-log-orig=15 is given,
the value store in the anyValue will be composed of:
   bit 0: no logging
   bit 1-15: server id
   bit 16-32: null

To make sure current setups don't break, --ndb-log-orig will default
to storing the server id as 32bit integer.

* --ndb-log-orig : default, store server id 32bit
* --ndb-log-orig=X : store server id as Xbit integer

There should also be an error if the server id is impossible when
--ndb-log-orig is given a value. mysqld should exit.

One problem remeans, I think, when Master and Slave have different
--ndb-log-orig options set.

How to repeat:
.

Design thoughts

Attachment: anyValue.txt (text/plain), 9.70 KiB.

Proposed patch allowing user overload of AnyValue data

Attachment: bug52305.patch (text/x-patch), 20.07 KiB.

A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

http://lists.mysql.com/commits/112732

3642 Frazer Clement 2010-07-02
Bug#52305 Customizing storage of server id inside the anyValue

Add support for server-id-bits to Ndb handler

Ndb handler is modified to support the server-id-bits mechanism as follows :
- Where server-id-bits is < 31, only those bits will be set on NdbApi operations created by the handler
- When receiving events for committed NdbApi operations, only the server-id-bits lsbs will be consulted to determine the source serverId (if any).
- When writing events to the Binlog, the user-data from the received AnyValue (if any) will be written into the Binlog event's ServerId. Only the bottom server-id-bits will be set to the event's ServerId (As received for replicated updates, or the server's own Id for non-replicated updates).

This mechanism allows :
- NdbApi applications to set the user-data portion of the AnyValue for their operations (bits server-id-bits to 30).
- NdbApi applications to listen to events and receive the user-data portion of their AnyValue.
- User-data portions of AnyValue to be recorded in the Binlog
- User-data portions of AnyValue to be tagged onto replicated updates in the slave
- User-data portions of AnyValue to be recorded in the Slave's Binlog....
- Normal 'no-logging', 'log-slave-updates', 'replication-loop-detection' and 'ndb_apply_status' mechanism work as expected.

The default value of server-id-bits is 32, which causes no change in the current behaviour of the Ndb handler w.r.t. AnyValue and ServerIds.

Note that in future it may be useful to offer a configuration where the userdata portion of the AnyValue is ignored within a Cluster, but userdata is *not* recorded in the ServerId field of Binlogged events.
@ sql/ha_ndbcluster.cc
Change slave-applying-replicated-event AnyValue setting code to set the AnyValue to the received unmasked_server_id rather than the masked server_id. This causes extra user-data to pass from Master-Binlog to the slave Cluster.
@ sql/ha_ndbcluster_binlog.cc
Add AnyValue get/set/check methods which use the server-id-bits mask.
Change most get/set/check of AnyValue to use these.

Modify replicated-schema-event AnyValue setting code to set the AnyValue to the unmasked value received from the Binlog. This allows Master Binlog user data to pass into the Cluster. Note that for schema events this is not currently used.
@ sql/ha_ndbcluster_binlog.h
Move #defines to implementation
Add AnyValue get/set/check function definitions.
@ storage/ndb/tools/restore/consumer_restore.cpp
Modify AnyValue-for-nologging #define used in ndb_restore.

A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

http://lists.mysql.com/commits/112733

3643 Frazer Clement 2010-07-02
Bug#52305 Customizing storage of server id inside the anyValue

Test UserData part of AnyValue

Add a DBUG-only env variable to trigger MySQLD to set the user portion of AnyValue to all 1s.
Modify my.cnf for some AnyValue-sensitive testcases to set a 24-bit user portion which will be set to all 1s.
This gives coverage of the case(s) where the User portion is non-zero, checking that the system still functions as expected w.r.t. ServerId, NoLogging etc.

The existing repl_ndb_slave_lsu testcase is modified to also use the mysqlbinlog tool to look at the events recorded.
A new variant of it called rpl_ndb_slave_lsu_anyval runs with the top 24 bits set to 1 and server-id-bits set to 7. The testcase ensures that :
- Server-ids are correctly extracted from the composite serverid
- Composite serverids are binlogged
- Composite serverids pass through the slave and can be binlogged
- mysqlbinlog can be used to see composite server id values.
@ mysql-test/suite/rpl_ndb/r/rpl_ndb_slave_lsu.result
Modified rpl_ndb_slave_lsu result file due to mysqlbinlog output inclusion.
@ mysql-test/suite/rpl_ndb/r/rpl_ndb_slave_lsu_anyval.result
Result file for new testcase
@ mysql-test/suite/rpl_ndb/t/rpl_ndb_get_binlog_events.inc
New include file for deterministically outputing interesting parts of mysqlbinlog file
@ mysql-test/suite/rpl_ndb/t/rpl_ndb_multi_binlog_update.inc
Modify file to also show contents from mysqlbinlog p.o.v
@ mysql-test/suite/rpl_ndb/t/rpl_ndb_slave_lsu_anyval.cnf
'Wrapper' around existing rpl_ndb_slave_lsu to give coverage of non-default server-id-bits setting behvaiour where updates have user-data set.
@ mysql-test/suite/rpl_ndb/t/rpl_ndb_slave_lsu_anyval.test
Test file checks that we have debug on (for the 'error injection') and runs rpl_ndb_slave_lsu
@ sql/ha_ndbcluster.cc
Add 'error injection' based on ENV var in debug only.
@ sql/ha_ndbcluster_binlog.cc
Add Error injection mechanism.
@ sql/ha_ndbcluster_binlog.h
Add Error injection mechanism

A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

http://lists.mysql.com/commits/112734

3641 Frazer Clement 2010-07-02
Bug#52305 : Customizing storage of server id inside the anyValue

Add server-id-bits option to MySQLD

A general server-id-bits option is added to MySQLD initialisation options.
This option indicates the number of lsbs within the 32-bit ServerId which actually contain the server-id.
Indicating that the actual server-id uses < 32 bits allows some of the remaining bits to be used for other purposes. When extracting the actual server-id for purposes such as replication-loop-detection, the server will ignore the irrelevant bits.
If the configured server-id cannot be expressed in server-id-bits then the MySQLD will refuse to start.
The server-id bits option is used mask out the irrelevant bits of the serverid in the IO and SQL threads when deciding whether an event should be ignored based on serverid.
The event's serverid is used to set the slave thread's thd->server_id in the normal way, and a new thd variable, unmasked_server_id carries the full 32-bits of the received server_id.
In cases where server_id_bits is < 32, handlers can look at thd->unmasked_server_id to get the non-server-id data associated with the event.

The mysqlbinlog tool gets a new option --server-id-bits which controls how it interprets the server ids in the Binlogs it reads. As with MySQLD this defaults to 32. Where the Binlog was written by a MySQLD with server-id-bits < 32 and some useful info in the msbs, running mysqlbinlog with server-id-bits of 32 allows this data to be seen.
@ client/mysqlbinlog.cc
Add server-id-bits option to mysqlbinlog tool
@ sql/log_event.cc
Modify log_event constructors to set unmasked_server_id.
Modify log_event from-data constructor to set server_id using server-id-bits mask.
@ sql/log_event.h
Add unmasked_server_id field to log_event base class.
@ sql/mysqld.cc
Add server-id-bits option to MySQLD.
Cross-verify server-id and server-id-bits in MySQLD startup.
Generate server_id_mask from server-id-bits.
@ sql/set_var.cc
Expose server-id-bits as a const system variable.
@ sql/slave.cc
Modify Slave SQL thread to set thd's unmasked_server_id from log_event's unmasked_server_id before executing.

Modify Slave IO thread to check masked server_id when performing same-server-id checks.
@ sql/slave.h
Import external server_id_mask to slave code.
@ sql/sql_class.cc
Initialise unmasked_server_id to server_id in the thd.
@ sql/sql_class.h
Add unmasked_server_id to thd class to enable handler code to access it while processing replicated events.

Pushed into 5.1.47-ndb-7.0.17 (revid:frazer@mysql.com-20100702110148-58ys2v0r3jgb7na1) (version source revid:frazer@mysql.com-20100702110148-58ys2v0r3jgb7na1) (merge vers: 5.1.47-ndb-7.0.17) (pib:16)

Above commits on 7.0 as a single patch

Attachment: bug52305.patch (text/x-patch), 107.20 KiB.

Pushed to 7.0.17 and 7.1.6.

Documented in the NDB-7.0.17 and 7.1.6 changelogs as follows:

        Added the --server-id-bits option for mysqld and mysqlbinlog.

        For mysqld, the --server-id-bits option indicates the number of
        least significant bits within the 32-bit server ID which
        actually identify the server. Indicating that the server ID uses
        less than 32 bits allows some of the remaining bits to be used
        for other purposes by NDB API applications using the Event API and
        NdbOperation::OperationOptions::AnyValue.

        For mysqlbinlog, the --server-id-bits option tells mysqlbinlog
        how to interpret the server IDs in the binary log when the
        binary log was written by a mysqld having its server-id-bits set
        to less than the maximum.

See http://lists.mysql.com/commits/113415 for complete docs changes.

Closed.

A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/125054