Bug #105545 Can not serialize records with functional key
Description:
I use COM_BINLOG_DUMP to get ROW and FULL format binlog events from MySQL just like canal(https://github.com/alibaba/canal), and then parse it, but i meet a problem when a table with functional key in the special situation.
In the special situation, which a table with N column and M functional key( (N + M + 7) / 8 != (N + 7) / 8), we will fail to parse the row event according to the document(https://dev.mysql.com/doc/internals/en/rows-event.html)
Binlog::RowsEvent:
Payload
header:
if post_header_len == 6 {
4 table id
} else {
6 table id
}
2 flags
if version == 2 {
2 extra-data-length
string.var_len extra-data
}
body:
lenenc_int number of columns
string.var_len columns-present-bitmap1, length: (num of columns+7)/8
if UPDATE_ROWS_EVENTv1 or v2 {
string.var_len columns-present-bitmap2, length: (num of columns+7)/8
}
rows:
string.var_len nul-bitmap, length (bits set in 'columns-present-bitmap1'+7)/8
string.var_len value of each field as defined in table-map
if UPDATE_ROWS_EVENTv1 or v2 {
string.var_len nul-bitmap, length (bits set in 'columns-present-bitmap2'+7)/8
string.var_len value of each field as defined in table-map
}
... repeat rows until event-end
I try a lot of times to parse the row event, and I have found that something in the doc should be corrected or the MySQL source code should be modified.
How to repeat:
Create a table with 8 columns and 1 functional key which N = 8, M = 1 as mentioned above:
CREATE TABLE functional_key_test.t2 (c1 INT primary key, c2 INT, c3 varchar(10), c4 timestamp, c5 bigint, c6 float, c7 datetime, c8 bool, Index func_index ((ABS(c1))));
INSERT INTO functional_key_test.t2 (c1) VALUES (1);
We will get a TABLE_MAP_EVENT with column-count = 9 according to the doc(https://dev.mysql.com/doc/internals/en/table-map-event.html), it looks like the sum of number of real column and the number of virtual column.
And we will also get a WRITE_ROWS_EVENTv2 with the number of columns = 9 and length(nul-bitmap) = 16, when I parse the remaining event, I got a wrong result.
//BINLOG '
//WJmDYRN9ffeLUAAAABocCgAAAJMYAQAAAAEAE2Z1bmN0aW9uYWxfa2V5X3Rlc3QAAnQyAAkDAw8R
//CAQSAQMFHgAABAD2AAEBAAIBIVJnino=
//WJmDYRd9ffeLKwAAAEUcCgAAAJMYAQAAAAEACf/+9gEAAABhg5lYzfgqOw==
//'/*!*/;
//### INSERT INTO `functional_key_test`.`t2`
//### SET
//### @1=1 /* INT meta=0 nullable=0 is_null=0 */
//### @2=NULL /* INT meta=0 nullable=1 is_null=1 */
//### @3=NULL /* VARSTRING(30) meta=30 nullable=1 is_null=1 */
//### @4=1636014424 /* TIMESTAMP(0) meta=0 nullable=0 is_null=0 */
//### @5=NULL /* LONGINT meta=0 nullable=1 is_null=1 */
//### @6=NULL /* FLOAT meta=4 nullable=1 is_null=1 */
//### @7=NULL /* DATETIME(0) meta=0 nullable=1 is_null=1 */
//### @8=NULL /* TINYINT meta=0 nullable=1 is_null=1 */
Suggested fix:
The values of virtual columns are not stored in binlog at present, so I think the virtual column should not be counted in the TABLE_MAP_EVENT and WRITE_ROWS_EVENTv2,
if it has to be counted, the document should be modified. So i think there are 2 ways:
1. Correct the document and modify the calculation of nul-bitmap,
length (bits set in 'columns-present-bitmap1'+7)/8 should be changed to (number of real columns exclude virtual computed columns), the number of real columns can be calculate with the SQL, just like:
select count(*) from information_schema.columns where table_schema = 'functional_key_test' and table_name = 't1';
2. Modify the logic of record binlog
Portions of this website are copyright © 2001, 2002 The PHP Group
Page generated in 0.041 sec. using MySQL 8.4.7-u6-cloud
Timestamp references displayed by the system are UTC.
Content reproduced on this site is the property of the
respective copyright holders.
It is not reviewed in advance
by Oracle and does not necessarily represent the opinion of
Oracle or any other party.