Bug #77729 Cleanup procedure for Memory engine tables breaks GITD replica
Submitted: 15 Jul 2015 8:41 Modified: 7 Mar 2017 14:24
Reporter: Rumen Palov Email Updates:
Status: Verified Impact on me:
None 
Category:MySQL Server: Replication Severity:S2 (Serious)
Version:Distrib 5.6.16, for FreeBSD,5.6.35 OS:FreeBSD (tested only on freebsd)
Assigned to: CPU Architecture:Any

[15 Jul 2015 8:41] Rumen Palov
Description:
When you have table with Engine MEMORY working on replication master, mysqld injects "DELETE" statement in binary logs on first access query to this table. This insures consistent of data on replicating slaves. ( https://dev.mysql.com/doc/refman/5.6/en/memory-storage-engine.html / #MEMORY Tables and Replication  ) 

If replication is GTID ROW based, this inserted "DELETE" breaks replication.  Logged event is in STATEMENT format and do not generate correct SET GTID_NEXT statements in binary log.  

Binary Log snapshtop:
-------------------------------------------------------------------------------
SET @@SESSION.GTID_NEXT= '18c65f0b-1dae-11e4-89ba-00259087d9a8:6809'/*!*/;
# at 164462833
#150714 17:17:02 server id 663  end_log_pos 164462977 CRC32 0x7d3fbc90  Query   thread_id=183308        exec_time=3     error_code=0
use `information_schema`/*!*/;
SET TIMESTAMP=1436883422/*!*/;
SET @@session.sql_mode=0/*!*/;
SET @@session.auto_increment_increment=3, @@session.auto_increment_offset=3/*!*/;
/*!\C latin1 *//*!*/;
SET @@session.character_set_client=8,@@session.collation_connection=8,@@session.collation_server=8/*!*/;
DELETE FROM `schema_name`.`table_name`
/*!*/;
# at 164462977
#150714 17:17:02 server id 663  end_log_pos 164463113 CRC32 0x4b5397a3  Query   thread_id=183308        exec_time=4     error_code=0
SET TIMESTAMP=1436883422/*!*/;
DELETE FROM `schema_name`.`table_name`
/*!*/;
# at 164463113
-------------------------------------------------------------------------------

Error message in mysql log is as follow: 
-------------------------------------------------------------------------------
ERROR] Slave SQL: Error 'When @@SESSION.GTID_NEXT is set to a GTID, you must explicitly set it to a different value after a COMMIT or ROLLBACK. Please check GTID_NEXT variable manual page for detailed explanation. Current @@SESSION.GTID_NEXT is '18c65f0b-1dae-11e4-89ba-00259087d9a8:6809'.' on query. Default database: 'information_schema'. Query: 'DELETE FROM `nl_website_tickets`.`test_last_month`', Error_code: 1837
2015-07-14 17:17:08 91890 [Warning] Slave: When @@SESSION.GTID_NEXT is set to a GTID, you must explicitly set it to a different value after a COMMIT or ROLLBACK. Please check GTID_NEXT variable manual page for detailed explanation. Current @@SESSION.GTID_NEXT is '18c65f0b-1dae-11e4-89ba-00259087d9a8:6809'. Error_code: 1837
-------------------------------------------------------------------------------

Quick workaround - Inject empty gtid event, with GTID from error in mysql error log. 

How to repeat:
1) Setup replication with GTID enabled, ROW based.
2) Create memory table
3) Insert few rows
4) Restart Master mysqld
5) log in mysql console on master 
6) SELECT * from memory_table 
7) log in mysql console slave - replication is broken with messages from "Description" Section of this bug report

Suggested fix:
When mysql injects cleanup "DELETE"-S in binary log, they have to be in the same format as is binary log itself and configured replication type.
[16 Jul 2015 12:58] Umesh Shastry
Hello Rumen,

Thank you for the report.
I'm not seeing this issue on 5.6.25/5.6.27, I see you are using 5.6.16 which is very old and many bugs fixed since then. Please try GA version 5.6.25 and let us know if you are still having this issue along with conf files used and error log for further investigations.

If you can provide more information, feel free to add it to this bug and change the status back to 'Open'.

Thank you for your interest in MySQL.

Thanks,
Umesh
[16 Jul 2015 12:58] Umesh Shastry
5.6.25/27 test results

Attachment: 77729_5.6.25.results (application/octet-stream, text), 54.99 KiB.

[5 Aug 2015 10:11] Amar Jaiswal
I am still getting this issue in 5.6.25 as well as 5.6.26.
5.6.27 is still not released according to Mysql5.6 release-notes page.

Any updates...
[4 Apr 2016 16:28] monty solomon
Was this issue fixed as part of bug # 68525?
[7 Mar 2017 14:24] Umesh Shastry
Thank you for the feedback, and steps.