Design notes ------------ AnyValue mechanism ------------------ The AnyValue field is a per-DML-operation 'tag' used to attach some meta information to DML operations which can be retrieved via the NdbApi event API. It is 32 bits. At the Ndb level it can contain any value. If no value is supplied then it defaults to zero. MySQLD sets the AnyValue tag in the slave to tag DML operations with the ServerId of the MySQLD server where they were first Binlogged, which is received in the Binlog. The Binlog injector has AnyValue handling logic as follows : - if (AnyValue == 0) /* Treat as a local cluster originated change, Binlog it with our serverId */ else if (AnyValue > NDB_ANYVALUE_RESERVED) if (AnyValue == NDB_ANYVALUE_FOR_NOLOGGING) /* Do not log this operation */ else /* Unknown reserved AnyValue - warn and do not log */ else /* Non reserved, non zero anyValue, treat as originating serverId * Binlog operation iff we have log_slave_updates == On */ MySQLD has the following #defines regarding ANYVALUE : /* server id's with high bit set is reservered */ #define NDB_ANYVALUE_FOR_NOLOGGING 0xFFFFFFFF #define NDB_ANYVALUE_RESERVED 0x80000000 So the msb indicates that the rest of the bits are reserved for MySQLD 'special' usage. AnyValue ranges : 0 : Local cluster originated change, Binlog it 1 - 0x7fffffff : Replicated change from server X, Binlog iff log_slave_updates on 0x80000000 - 0xfffffffffe : Unused reserved values 0xffffffff : Event specifically not to be Binlogged 332 21 10 0 10987654321098765432109876543210 RSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS R = Reserved S = ServerId (0 == No serverId) Note that AnyValue is also used to tag the 'meta' updates to the ndb_schema internal table, indicating whether they originated locally or as part of replication. The current valid range for ServerIds is 1 -> 0x7fffffff, which is 2^31 -1 values. Note that this is less than 2^32 -1, and that using (for example) an IPv4 address as a ServerId would not work correctly if the first byte was > 0x7f, which is often the case. Feature request analysis ------------------------ The main issue in the bug seems to be that the operation tagging mechanism is currently exclusively for the use of MySQLD. The ability to tag operations with some meta information has lots of other applications, so ideally applications could add their own tagging information without affecting the behaviour of MySQLD. As the existing use of AnyValue places restrictions on the usable range of Server Ids, it sets a precedent for accepting a further such restriction. The bug proposes a mechanism where the significant digits of the ServerId are specified by configuration. MySQLD would use only those binary digits, and ignore the settings of other bits, freeing the other bits for use by applications. Having a fully configurable significant range gives flexible tradeoffs between ServerId range and the number of bits available for application purposes. Example : For example, with 8 bits reserved for ServerId identification (Giving 255 ServerIds), the AnyValue word would look like : 332 21 10 0 10987654321098765432109876543210 RAAAAAAAAAAAAAAAAAAAAAAASSSSSSSS R = Reserved (Bit 31) A = Application usage (23 bits 8->31) S = ServerId (8 bits 0->7) To understand this AnyValue, the following algorithm could be used : if (AnyValue && 0x80000000) { /* Reserved bit handling - no application/serverId info * carried */ } else { Uint32 serverId = AnyValue & 0xff; Uint32 appInfo = (AnyValue >> 8) & 0x7fffff; ... } Assuming that : - all existing ServerIds fit within the truncated range - Application Specific bits are not set to non-zero values until after all MySQLDs are upgraded Then : - When a slave MySQLD is upgraded, it will contine to set the same AnyValue values until the Application bits become non-zero. - When a binlogging MySQLD is upgraded, it will continue to correctly interpret the shorter ServerIds. This is preferable to a scheme rearranging the order of the bits, or shifting the ServerId as it is easier to deal with online upgrade issues. Note that this initial scheme continues the assumption that : - The reserved bit indicates that the rest of the bits are open to future reinterpretation (and cannot contain Application info or ServerId info) There is currently only one specific usage of the'Reserved' bit - the NO_LOGGING anyvalue where all ServerId bits are set. It is not clear in general whether it's valid to state that there is no need for application information in the AnyValue when MySQLD tags the operation as NO_LOGGING. It seems possible that some use case may require to carry application info with an operation tagged as NO_LOGGING. Options to support this include : 1) NO_LOGGING is inferred from the combination of the Reserved bit and all ServerIds being set. Application info bits are ignored. Upgrade implication : Old injector : NO_LOGGING value changes from 0xffffffff to a range of values - non upgraded Binlog injectors would treat as a bad reserved code and warn, but would correctly not-log the value. Old slave : 0xffffffff will map to NO_LOGGING with the new scheme. 2) NO_LOGGING is inferred from all the ServerId bits being set. This removes one potential serverId from the range, leaving 2^s -2 values for s bits. Application info bits are ignored. Upgrade implication : Old injector : NO_LOGGING value changes from 0xffffffff to a value without the reserved bit set - non upgraded Binlog injectors would treat as a server Id. Old slave : 0xffffffff will map to a reserved code (no logging). Option 1) preferred as more similar to current mechanism, avoids further loss of ServerId range and allows application info to be transported with reserved info. Note that this reduces the range of reserved values down to be dependent on the configurable ServerId range. For this reason, a lower bound on the number of ServerId bit should be chosen to avoid this reserved range being too small. Perhaps 7 bits as a lower bound is reasonable? It gives scope for 2^7 -1 = 127 further reserved codes or 6.5 bits for MySQL future use. 2^7 -1 = 127 serverIds. 2^(32 - 1 - 7) = At most 2^24 Application Info bits. Configuration ------------- The bug report suggests using extra information in the --ndb-log-orig server option. However, this option controls which ndb_apply_status updates are generated by the Binlog injector, not how the AnyValue is used. Controlling the significant bits used in the AnyValue seems orthogonal to whether the ndb-log-orig option is set, so a separate parameter should be created. Note that the interpretation of the AnyValue data is a cluster-wide issue as all attached APIs receive the same events. However, currently interpretation occurs above the Ndb level, so configuration will be done via MySQLD. It remains the user responsibility to : 1) Ensure all MySQLDs are upgraded to support AnyValue-with-application-info 2) Ensure all MySQLDs are configured with the same number of significant ServerId bits 3) Ensure their applications are configured with the same number of significant ServerId bits Proposed parameter : ndb-serverid-bits Default and maximum value is 31 as the msb is 'reserved' Minimum value is 7 so that we there is some scope for future 'reserved' codes. Controls the number of bits used to represent MySQLD ServerIds in this Cluster. All MySQLDs attached to this Cluster and replicating to this Cluster must have ServerIds representable in this number of bits. Bits from ndb-serverid-bits to 30 in the AnyValue will be ignored by attached MySQLDs. ServerIds configured or received in Binlog entries outwith this representable range will generate warnings and be 'truncated'. Implementation -------------- 1) Add ndb-serverid-bits configuration parameter 2) Add methods to set/get ServerId and NO_LOGGING value to/from AnyValue using configuration etc. getServerId(AnyValue) { return AnyValue & ((1 < ndb_serverid_bits) -1); } setServerId(AnyValue&, ServerId){ Mask = ((1 < ndb_serverid_bits) -1); AnyValue &= ~Mask; AnyValue |= (ServerId & Mask); } isServerIdInRange(serverId) { return serverId < (1 << ndb_serverid_bits); } isReservedValue(AnyValue) { return (serverId & 0x80000000) != 0; } const Uint32 RESERVED_CODES_MASK = (1 << 7) -1; const Uint32 NDB_VALUE_FOR_NO_LOGGING = 0x7f; isNoLoggingValue(AnyValue) { return (isReservedValue(AnyValue) && (AnyValue & RESERVED_CODES_MASK) == NDB_VALUE_FOR_NO_LOGGING); }; setNoLoggingValue(AnyValue) { AnyValue |= 0x80000000; AnyValue |= NDB_VALUE_FOR_NO_LOGGING }; 3) Modify existing code to use new methods : ha_ndbcluster.cc::eventSetAnyValue() ha_ndbcluster_binlog.cc::ndbcluster_log_schema_op() ha_ndbcluster_binlog.cc::ndb_binlog_query() ha_ndbcluster_binlog.cc::ndb_binlog_thread_handle_schema_event() ha_ndbcluster_binlog.cc::ndb_binlog_thread_handle_data_event() Testing ------- 1) Test that existing replication, serverId etc testcases pass with default config 2) Test boundary conditions of config parameter 3) Test that a) locally sourced, b) replicated, c) Nologging scenarios work with randomised application info for i) Data events, ii) Schema events 3) Test that existing testcases pass with randomised 'application data' 4) Test upgrade scenarios : i) Upgrade with default configuration - no effect ii) Mixed configurations have expected effect of warnings for NO_LOGGING.