Bug #84153 ASSERT "!contains_gtid(gtid)" rpl_gtid_owned.cc:94 Owned_gtids::add_gtid_owner
Submitted: 9 Dec 2016 19:05 Modified: 9 Jun 2017 9:57
Reporter: Narendra Singh Chauhan Email Updates:
Status: Closed Impact on me:
None 
Category:MySQL Server: Replication Severity:S2 (Serious)
Version:5.7.18 OS:Any
Assigned to: CPU Architecture:Any

[9 Dec 2016 19:05] Narendra Singh Chauhan
Description:
Scenario:- What happens when a existing GTID_NEXT transaction gets conflicting GTID generated by server?
Expected Output:- Either COMMIT should fail or something like ER1778.
Actual Output:- Assertion failure: sql/rpl_gtid_owned.cc:85: enum_return_status Owned_gtids::add_gtid_owner(const Gtid&, my_thread_id): Assertion `!contains_gtid(gtid)' failed.

Example:-
Assume we have gtid_executed='aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa:1', then:
1. client1: set gtid_next='aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa:2'; begin; insert into t1 values (1);
2. client2: CREATE TEMPORARY TABLE tt (a INT); \r (disconnect/reconnect)  << Note* In 8.0.1 we can see ASSERT here.
3. client1: COMMIT;  << Note* In 5.7.18 we can see ASSERT here.

mysqld.2.err:-
===============
mysqld: /Narendra/mysql_work/git_repo/mysql-5.7/sql/rpl_gtid_owned.cc:85: enum_return_status Owned_gtids::add_gtid_owner(const Gtid&, my_thread_id): Assertion `!contains_gtid(gtid)' failed.
18:54:33 UTC - mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
Attempting to collect some information that could help diagnose the problem.
As this is a crash and something is definitely wrong, the information
collection process might fail.

key_buffer_size=1048576
read_buffer_size=131072
max_used_connections=7
max_threads=151
thread_count=10
connection_count=7
It is possible that mysqld could use up to 
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 61087 K  bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0x7f10980117b0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 7f11180eddd0 thread_stack 0x40000
/Narendra/mysql_work/git_repo/mysql-5.7/install/bin/mysqld(my_print_stacktrace+0x38)[0x18fd604]
/Narendra/mysql_work/git_repo/mysql-5.7/install/bin/mysqld(handle_fatal_signal+0x40a)[0xf23395]
/lib/x86_64-linux-gnu/libpthread.so.0(+0xfcb0)[0x7f111feefcb0]
/lib/x86_64-linux-gnu/libc.so.6(gsignal+0x35)[0x7f111f332035]
/lib/x86_64-linux-gnu/libc.so.6(abort+0x17b)[0x7f111f33579b]
/lib/x86_64-linux-gnu/libc.so.6(+0x2ee1e)[0x7f111f32ae1e]
/lib/x86_64-linux-gnu/libc.so.6(+0x2eec2)[0x7f111f32aec2]
/Narendra/mysql_work/git_repo/mysql-5.7/install/bin/mysqld(_ZN11Owned_gtids14add_gtid_ownerERK4Gtidj+0x69)[0x181a08b]
/Narendra/mysql_work/git_repo/mysql-5.7/install/bin/mysqld(_ZN10Gtid_state17acquire_ownershipEP3THDRK4Gtid+0x13f)[0x1815f51]
/Narendra/mysql_work/git_repo/mysql-5.7/install/bin/mysqld(_ZN10Gtid_state23generate_automatic_gtidEP3THDixPi+0x24f)[0x18172eb]
/Narendra/mysql_work/git_repo/mysql-5.7/install/bin/mysqld(_ZN13MYSQL_BIN_LOG37assign_automatic_gtids_to_flush_groupEP3THD+0x10c)[0x185a384]
/Narendra/mysql_work/git_repo/mysql-5.7/install/bin/mysqld(_ZN13MYSQL_BIN_LOG25process_flush_stage_queueEPyPbPP3THD+0x11d)[0x186c167]
/Narendra/mysql_work/git_repo/mysql-5.7/install/bin/mysqld(_ZN13MYSQL_BIN_LOG14ordered_commitEP3THDbb+0x4cb)[0x186d6eb]
/Narendra/mysql_work/git_repo/mysql-5.7/install/bin/mysqld(_ZN13MYSQL_BIN_LOG6commitEP3THDb+0xcea)[0x186bde4]
/Narendra/mysql_work/git_repo/mysql-5.7/install/bin/mysqld(_Z22close_temporary_tablesP3THD+0x9b0)[0x152baf1]
/Narendra/mysql_work/git_repo/mysql-5.7/install/bin/mysqld(_ZN3THD7cleanupEv+0x25c)[0x1551370]
/Narendra/mysql_work/git_repo/mysql-5.7/install/bin/mysqld(_ZN3THD17release_resourcesEv+0x251)[0x1551681]
/Narendra/mysql_work/git_repo/mysql-5.7/install/bin/mysqld(handle_connection+0x23d)[0x16e6cff]
/Narendra/mysql_work/git_repo/mysql-5.7/install/bin/mysqld(pfs_spawn_thread+0x173)[0x192af9f]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x7e9a)[0x7f111fee7e9a]
/lib/x86_64-linux-gnu/libc.so.6(clone+0x6d)[0x7f111f3ef36d]

Trying to get some variables.
Some pointers may be invalid and cause the dump to abort.
Query (0): Connection ID (thread ID): 24
Status: KILL_CONNECTION

I'm attaching detailed MTR output with this bug.

How to repeat:
Details:-
=========
MySQL Version: 5.7.18

Steps to repro:-
=================
Run command:- ./mtr group_replication.group_replication_gtid_owned.test
Where,

$ cat group_replication_gtid_owned.test
--let $group_replication_group_name= aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa
--source ../inc/have_group_replication_plugin.inc
--source ../inc/group_replication.inc

--connect(server1_1, localhost,root,,,$MASTER_MYPORT)

--let $rpl_connection_name= server1
--source include/rpl_connection.inc
CREATE TABLE t1(a INT PRIMARY KEY);
--source include/rpl_sync.inc
--echo
--echo == START a transaction on one terminal ==
set gtid_next='aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa:4';
BEGIN;
INSERT INTO t1 VALUES (1);

--let $rpl_connection_name= server1_1
--source include/rpl_connection.inc
CREATE TEMPORARY TABLE tt (a INT);

--echo
--echo == ISSUE HERE: Upon disconnection/reconnection ==
--let $rpl_connection_name= server1
--source include/rpl_connection.inc
--disconnect server1_1

--echo
SHOW GLOBAL VARIABLES LIKE '%gtid%';
COMMIT;  ## Issue here in 5.7.18: Upon COMMIT
SHOW BINLOG EVENTS;

DROP TABLE t1;
--source ../inc/group_replication_end.inc
[9 Jun 2017 9:57] David Moss
Posted by developer:
 
Thank you for your feedback, this has been fixed in upcoming versions and the following was added to the 5.7.19 and 8.0.2 changelogs:

When an existing GTID_NEXT transaction was assigned a conflicting GTID by the server, Group Replication generated an assert upon detecting two transactions with same GTID. This was because Group Replication generates the GTID after conflict detection, which is later than with master/slave replication.  The fix relaxes some conditions to only be called when commit is done and a message has been added to alert you when a GTID has already been used.