Bug #68525 | Error "When GTID_NEXT is set to a GTID" ROW based replication | ||
---|---|---|---|
Submitted: | 28 Feb 2013 15:32 | Modified: | 7 Sep 2015 17:07 |
Reporter: | Nogueira Jesus | Email Updates: | |
Status: | Closed | Impact on me: | |
Category: | MySQL Server: Replication | Severity: | S1 (Critical) |
Version: | 5.6.10, 5.6.11 | OS: | Linux (64 bits) |
Assigned to: | CPU Architecture: | Any | |
Tags: | GTID_NEXT ROW BASED REPLICATION ERROR WHEN ROTATE LOGS |
[28 Feb 2013 15:32]
Nogueira Jesus
[28 Feb 2013 17:57]
Sveta Smirnova
Thank you for the report. Have you checked that table definition is same on both master and slave? Please send us output of SHOW CREATE TABLE taken on each of these servers.
[28 Feb 2013 19:08]
Nogueira Jesus
Yes, checked, its identical.
[28 Feb 2013 19:15]
Nogueira Jesus
When making more tests we get another error, now is not only when rotate logs: When GTID_NEXT is set to a GTID, you must explicitly set it again after a COMMIT or ROLLBACK. If you see this error message in the slave SQL thread, it means that a table in the current transaction is transactional on the master and non-transactional on the slave. In a client connection, it means that you executed SET GTID_NEXT before a transaction and forgot to set GTID_NEXT to a different identifier or to 'AUTOMATIC' after COMMIT or ROLLBACK. Current GTID_NEXT is '76e884c2-7f8b-11e2-bd39-000c2904bdbb:156
[28 Feb 2013 19:47]
Sveta Smirnova
Thank you for the feedback. My version of MySQL complains about option gtid-enforce-consistency. Which exact MySQL package do you use?
[28 Feb 2013 19:52]
Sveta Smirnova
No feedback needed. Verified as described with option enforce-gtid-consistency
[28 Feb 2013 19:56]
Sveta Smirnova
Workaround: use transactional storage engine InnoDB.
[28 Feb 2013 19:56]
Sveta Smirnova
test case for MTR
Attachment: rpl_bug68525.test (application/octet-stream, text), 1.06 KiB.
[28 Feb 2013 19:57]
Sveta Smirnova
master option file, slave uses same option file
Attachment: rpl_bug68525-master.opt (application/octet-stream, text), 267 bytes.
[28 Feb 2013 20:50]
Nogueira Jesus
I cannot move to InnoDB, its not only a one or two tables, its thousand´s of tables in this format, and the best storage engine we can use is MyISAM, InnoDB its out of question, we already tested in past InnoDB for our systems, but its not satisfatory for our activities/types of query´s most used´s, and all our systems are designed to run over MYISAM. Im just start to replicate for backup porpouses. I´ve tried run this pl script, but i´ve installed mysql from rpm and not have a source. So, i´ve run that query and the error come up again. When GTID_NEXT is set to a GTID, you must explicitly set it again after a COMMIT or ROLLBACK. If you see this error message in the slave SQL thread, it means that a table in the current transaction is transactional on the master and non-transactional on the slave. In a client connection, it means that you executed SET GTID_NEXT before a transaction and forgot to set GTID_NEXT to a different identifier or to 'AUTOMATIC' after COMMIT or ROLLBACK. Current GTID_NEXT is '76e884c2-7f8b-11e2-bd39-000c2904bdbb:17'
[28 Feb 2013 20:54]
Nogueira Jesus
By the way, we changed max_binlog_size to 1024 too and run that query, its confirmed, the problem occurs again, 7 binlog files are been created in this test.
[15 Oct 2013 20:47]
martin fuxa
maybe I have same experience with Mysql 5.6.14 (both master and slave), but with another error msg. also during log rotation. and yes there are some MyISAM tables, right this wa_x_uzivatele (see attached log) error 2013-10-14 12:27:47 9900 [ERROR] Slave SQL: When @@SESSION.GTID_NEXT is set to a GTID, you must explicitly set it to a different value after a COMMIT or ROLLBACK. Please check GTID_NEXT variable manual page for detailed explanation. Current @@SESSION.GTID_NEXT is '1be2af64-ce18-11e2-bd6e-00151743f04a:5644022'. Error_code: 1837 2013-10-14 12:27:47 9900 [Warning] Slave: When @@SESSION.GTID_NEXT is set to a GTID, you must explicitly set it to a different value after a COMMIT or ROLLBACK. Please check GTID_NEXT variable manual page for detailed explanation. Current @@SESSION.GTID_NEXT is '1be2af64-ce18-11e2-bd6e-00151743f04a:5644022'. Error_code: 1837 2013-10-14 12:27:47 9900 [ERROR] Error running query, slave SQL thread aborted. Fix the problem, and restart the slave SQL thread with "SLAVE START". We stopped at log 'mysql-bin.000008' position 629122030 decoded relay-bin attached end of previous with failed GTID mysqlbinlog -v --base64-output=DECODE-ROWS /var/lib/mysql/h7-relay-bin.000023 and next one mysqlbinlog -v --base64-output=DECODE-ROWS /var/lib/mysql/h7-relay-bin.000024 my conf: max_relay_log_size = 600M binlog_format=ROW innodb_file_per_table innodb_flush_log_at_trx_commit = 2 gtid-mode=ON enforce-gtid-consistency=true log-slave-updates=true read_only=ON relay_log_info_repository = TABLE relay_log_recovery = ON
[15 Oct 2013 20:48]
martin fuxa
decoded relay log with comments
Attachment: h7-repl-2243.log (text/x-log), 7.77 KiB.
[3 Jan 2014 15:23]
James Esslinger
Has there been any resolution to this issue besides the suggestion of switching to InnoDB?
[9 Jan 2014 16:15]
Nuno Carvalho
No, there is no resolution yet to this MyISAM-only bug.
[28 Jan 2014 10:30]
Arvind Sharma
I get a similar error even with all tables being InnoDB. I have a MySQL 5.6.15 SLAVE on CentOS 6.5 used just for backups. On some days, during nightly backup (Percona Xtrabackup) when SQL_THREAD is stopped briefly, the following error is thrown. The result is missing transactions and inconsistent state of the Slave. 2014-01-27 22:48:37 26644 [ERROR] Error reading packet from server: Lost connection to MySQL server during query ( server_errno=2013) 2014-01-27 22:48:37 26644 [Note] Slave I/O thread: Failed reading log event, reconnecting to retry, log 'mysql-bin.006225' at position 7557419 2014-01-27 22:48:37 26644 [Warning] Storing MySQL user name or password information in the master info repository is not secure and is therefore not recommended. Please consider using the USER and PASSWORD connection options for START SLAVE; see the 'START SLAVE Syntax' in the MySQL Manual for more information. 2014-01-27 22:50:11 26644 [ERROR] Error reading packet from server: Lost connection to MySQL server during query ( server_errno=2013) 2014-01-27 22:50:11 26644 [Note] Slave I/O thread: Failed reading log event, reconnecting to retry, log 'mysql-bin.006225' at position 7532885 2014-01-27 22:50:11 26644 [Warning] Storing MySQL user name or password information in the master info repository is not secure and is therefore not recommended. Please consider using the USER and PASSWORD connection options for START SLAVE; see the 'START SLAVE Syntax' in the MySQL Manual for more information. 2014-01-27 22:51:46 26644 [ERROR] Error reading packet from server: Lost connection to MySQL server during query ( server_errno=2013) 2014-01-27 22:51:46 26644 [Note] Slave I/O thread: Failed reading log event, reconnecting to retry, log 'mysql-bin.006225' at position 7500173 2014-01-27 22:51:46 26644 [Warning] Storing MySQL user name or password information in the master info repository is not secure and is therefore not recommended. Please consider using the USER and PASSWORD connection options for START SLAVE; see the 'START SLAVE Syntax' in the MySQL Manual for more information. 2014-01-27 22:53:10 26644 [ERROR] Error reading packet from server: Lost connection to MySQL server during query ( server_errno=2013) 2014-01-27 22:53:10 26644 [Note] Slave I/O thread: Failed reading log event, reconnecting to retry, log 'mysql-bin.006225' at position 7516529 2014-01-27 22:53:10 26644 [Warning] Storing MySQL user name or password information in the master info repository is not secure and is therefore not recommended. Please consider using the USER and PASSWORD connection options for START SLAVE; see the 'START SLAVE Syntax' in the MySQL Manual for more information. 2014-01-27 22:55:10 26644 [ERROR] Error reading packet from server: Lost connection to MySQL server during query ( server_errno=2013) 2014-01-27 22:55:10 26644 [Note] Slave I/O thread: Failed reading log event, reconnecting to retry, log 'mysql-bin.006225' at position 7549241 2014-01-27 22:55:10 26644 [Warning] Storing MySQL user name or password information in the master info repository is not secure and is therefore not recommended. Please consider using the USER and PASSWORD connection options for START SLAVE; see the 'START SLAVE Syntax' in the MySQL Manual for more information. 2014-01-27 22:56:21 26644 [ERROR] Error reading packet from server: Lost connection to MySQL server during query ( server_errno=2013) 2014-01-27 22:56:21 26644 [Note] Slave I/O thread: Failed reading log event, reconnecting to retry, log 'mysql-bin.006225' at position 7508351 2014-01-27 22:56:21 26644 [Warning] Storing MySQL user name or password information in the master info repository is not secure and is therefore not recommended. Please consider using the USER and PASSWORD connection options for START SLAVE; see the 'START SLAVE Syntax' in the MySQL Manual for more information. 2014-01-27 22:57:55 26644 [ERROR] Error reading packet from server: Lost connection to MySQL server during query ( server_errno=2013) 2014-01-27 22:57:55 26644 [Note] Slave I/O thread: Failed reading log event, reconnecting to retry, log 'mysql-bin.006225' at position 7516529 2014-01-27 22:57:55 26644 [Warning] Storing MySQL user name or password information in the master info repository is not secure and is therefore not recommended. Please consider using the USER and PASSWORD connection options for START SLAVE; see the 'START SLAVE Syntax' in the MySQL Manual for more information. 2014-01-27 23:51:32 26644 [ERROR] Slave SQL: When @@SESSION.GTID_NEXT is set to a GTID, you must explicitly set it to a different value after a COMMIT or ROLLBACK. Please check GTID_NEXT variable manual page for detailed explanation. Current @@SESSION.GTID_NEXT is '949f7069-1c4d-11e3-cb23-005056ac66f0:46423910'. Error_code: 1837 2014-01-27 23:51:32 26644 [Warning] Slave: When @@SESSION.GTID_NEXT is set to a GTID, you must explicitly set it to a different value after a COMMIT or ROLLBACK. Please check GTID_NEXT variable manual page for detailed explanation. Current @@SESSION.GTID_NEXT is '949f7069-1c4d-11e3-cb23-005056ac66f0:46423910'. Error_code: 1837 2014-01-27 23:51:32 26644 [ERROR] Error running query, slave SQL thread aborted. Fix the problem, and restart the slave SQL thread with "SLAVE START". We stopped at log 'mysql-bin.006225' position 7377299
[11 Feb 2014 12:10]
lalit Choudhary
using 5.6.15,checked slave status SQL_THREAD stop running. Last_error 'When @@SESSION.GTID_NEXT is set to a GTID, you must explicitly set it to a different value after a COMMIT or ROLLBACK. Please check GTID_NEXT variable manual page for detailed explanation. Current @@SESSION.GTID_NEXT is ''4r2ad77f-697e-11g3-b2c3-c80aa9f67dc4:160058197''.' -restarted slave now replication in sync.
[21 Apr 2014 9:09]
Shahriyar Rzayev
I have a similar error full flow is stated below: Get a full dump using mysqldump from MySQL 5.6.10 with following commmand: mysqldump -u root -p --all-databases --single-transaction --flush-logs --set-gtid-purged=OFF > /home/new_backups/apa_full_backup9.sql After getting dump i prepare 2 Virtual Machines with CEntos 6.5 with MySQL 5.6.17 installed from official repo. Imported dump in BOTH MySQL servers and after import set a replication between them as master->slave. After a while a got a: " When @@SESSION.GTID_NEXT is set to a GTID, you must explicitly set it to a different value after a COMMIT or ROLLBACK. Please check GTID_NEXT variable manual page for detailed explanation. Current @@SESSION.GTID_NEXT is" I don't know if it is good practice or not but after doing: stop slave; start slave; in other words after restarting slave it is solved.
[7 May 2014 12:56]
Van Stokes
OS: Ubuntu 14.04 LTS (x64) MySQL Server: 5.6.16 (x64) Ubuntu distro Replication Topology: Circular (four masters) Same issue with us. 2014-05-07 08:21:42 8313 [ERROR] Slave SQL: When @@SESSION.GTID_NEXT is set to a GTID, you must explicitly set it to a different value after a COMMIT or ROLLBACK. Please check GTID_NEXT variable manual page for detailed explanation. Current @@SESSION.GTID_NEXT is '819c985c-d384-11e3-a621-00259002979a:90332'. Error_code: 1837 2014-05-07 08:21:42 8313 [Warning] Slave: When @@SESSION.GTID_NEXT is set to a GTID, you must explicitly set it to a different value after a COMMIT or ROLLBACK. Please check GTID_NEXT variable manual page for detailed explanation. Current @@SESSION.GTID_NEXT is '819c985c-d384-11e3-a621-00259002979a:90332'. Error_code: 1837 2014-05-07 08:21:42 8313 [ERROR] Error running query, slave SQL thread aborted. Fix the problem, and restart the slave SQL thread with "SLAVE START". We stopped at log 'master-bin.000090' position 984533096 2014-05-07 08:38:09 8313 [Note] Slave I/O thread exiting, read up to log 'master-bin.000091', position 705242459 2014-05-07 08:38:25 8313 [Warning] Storing MySQL user name or password information in the master info repository is not secure and is therefore not recommended. Please consider using the USER and PASSWORD connection options for START SLAVE; see the 'START SLAVE Syntax' in the MySQL Manual for more information. 2014-05-07 08:38:25 8313 [Note] Slave I/O thread: connected to master 'rs_1002@atl-mysql02.econocaribe.com:3306',replication started in log 'master-bin.000091' at position 705242459 2014-05-07 08:38:25 8313 [Note] Slave SQL thread initialized, starting replication in log 'master-bin.000090' at position 984533096, relay log './slave-relay-bin.000191' position: 491 2014-05-07 08:41:08 8313 [ERROR] Slave SQL: When @@SESSION.GTID_NEXT is set to a GTID, you must explicitly set it to a different value after a COMMIT or ROLLBACK. Please check GTID_NEXT variable manual page for detailed explanation. Current @@SESSION.GTID_NEXT is '819c985c-d384-11e3-a621-00259002979a:91145'. Error_code: 1837 2014-05-07 08:41:08 8313 [Warning] Slave: When @@SESSION.GTID_NEXT is set to a GTID, you must explicitly set it to a different value after a COMMIT or ROLLBACK. Please check GTID_NEXT variable manual page for detailed explanation. Current @@SESSION.GTID_NEXT is '819c985c-d384-11e3-a621-00259002979a:91145'. Error_code: 1837 2014-05-07 08:41:08 8313 [ERROR] Error running query, slave SQL thread aborted. Fix the problem, and restart the slave SQL thread with "SLAVE START". We stopped at log 'master-bin.000091' position 702641450
[29 May 2014 12:18]
Van Stokes
OS: Ubuntu 14.04 LTS (x64) and Windows Server R2008 (x64) MySQL: 5.6.17 (x64) (Ubuntu distro) and 5.6.17 (x64) (MySQL distro for Windows) This problem happens very frequently. Can we get an update on this!? Furthermore, we found that after this error occurs and you attempt to perform: SLAVE STOP; and then a SLAVE START; the client hangs while trying to execute the SLAVE STOP; command. The mysql server must be restarted. Below is an example of the problem. NOTE how STOP SLAVE is hung at killing slave. Also notice how all other SLAVE commands are HUNG too (i.e. SHOW SLAVE STATUS). However, the other client connections seem unaffected and continue to function normally. mysql> show processlist\g +--------+-----------------+----------------------+-------------------+---------+---------+------------------------+-------------------+ | Id | User | Host | db | Command | Time | State | Info | +--------+-----------------+----------------------+-------------------+---------+---------+------------------------+-------------------+ | 3 | event_scheduler | localhost | NULL | Daemon | 1202642 | Waiting on empty queue | NULL | | 4 | webtools_eci | 127.0.0.1:54759 | eci_elite_prod | Sleep | 0 | | NULL | | 28 | webtools_eci | 127.0.0.1:54783 | eci_elite_prod | Sleep | 197 | | NULL | | 30 | webtools_eci | 127.0.0.1:54785 | eci_elite_prod | Sleep | 0 | | NULL | | 33 | webtools_eci | 127.0.0.1:54788 | eci_elite_prod | Sleep | 197 | | NULL | | 34 | webtools_eci | 127.0.0.1:54789 | eci_elite_prod | Sleep | 11 | | NULL | | 231 | webtools_eci | 127.0.0.1:54904 | eci_elite_prod | Sleep | 197 | | NULL | | 232 | webtools_eci | 127.0.0.1:54905 | eci_elite_prod | Sleep | 11 | | NULL | | 233 | webtools_eci | 127.0.0.1:54906 | eci_elite_prod | Sleep | 11 | | NULL | | 827 | webtools_eci | 127.0.0.1:55230 | eci_elite_prod | Sleep | 197 | | NULL | | 828 | webtools_eci | 127.0.0.1:55231 | eci_elite_prod | Sleep | 11 | | NULL | | 829 | webtools_eci | 127.0.0.1:55232 | eci_elite_prod | Sleep | 2067 | | NULL | | 830 | webtools_eci | 127.0.0.1:55233 | eci_elite_prod | Sleep | 11 | | NULL | | 831 | webtools_eci | 127.0.0.1:55234 | eci_elite_prod | Sleep | 11 | | NULL | | 125711 | webtools_eci | 127.0.0.1:50131 | eci_elite_prod | Sleep | 197 | | NULL | | 183624 | webtools_eci | 127.0.0.1:59254 | eci_elite_prod | Sleep | 0 | | NULL | | 449700 | webtools_eci | 127.0.0.1:61726 | eci_edi_eculine | Sleep | 911 | | NULL | | 544164 | webtools_eci | 127.0.0.1:55293 | eci_edi_eculine | Sleep | 207 | | NULL | | 544166 | webtools_eci | 127.0.0.1:55295 | eci_edi_eculine | Sleep | 911 | | NULL | | 579937 | monyog | 172.27.101.243:43145 | NULL | Query | 244 | init | SHOW SLAVE STATUS | | 579953 | monyog | 172.27.101.243:43237 | NULL | Sleep | 519 | | NULL | | 579960 | monyog | 172.27.101.243:43274 | NULL | Sleep | 3 | | NULL | | 649492 | webtools_eci | 127.0.0.1:50899 | eci_edi_eculine | Sleep | 7122 | | NULL | | 650546 | webtools_eci | 127.0.0.1:51482 | eci_webtools_prod | Sleep | 11 | | NULL | | 650551 | webtools_eci | 127.0.0.1:51486 | eci_edi_eculine | Sleep | 11 | | NULL | | 664334 | webtools_eci | 127.0.0.1:58290 | eci_webtools_prod | Sleep | 207 | | NULL | | 664335 | webtools_eci | 127.0.0.1:58291 | eci_webtools_prod | Sleep | 11 | | NULL | | 664336 | webtools_eci | 127.0.0.1:58292 | eci_webtools_prod | Sleep | 7125 | | NULL | | 676062 | webtools_eci | 127.0.0.1:63523 | eci_webtools_prod | Sleep | 911 | | NULL | | 676063 | webtools_eci | 127.0.0.1:63524 | eci_webtools_prod | Sleep | 911 | | NULL | | 687816 | webtools_eci | 127.0.0.1:52132 | eci_edi_eculine | Sleep | 14 | | NULL | | 716727 | vstokes | 172.27.101.234:52425 | NULL | Query | 466 | Killing slave | STOP SLAVE | | 716728 | monyog | 172.27.101.243:42688 | NULL | Query | 465 | init | SHOW SLAVE STATUS | | 716729 | monyog | 172.27.101.243:42693 | NULL | Query | 465 | init | SHOW SLAVE STATUS | | 716730 | monyog | 172.27.101.243:42701 | NULL | Query | 460 | init | SHOW SLAVE STATUS | | 716731 | vstokes | 172.27.101.234:52446 | NULL | Query | 459 | init | SHOW SLAVE STATUS | | 716871 | vstokes | 172.27.101.234:52625 | NULL | Sleep | 50 | | NULL | | 716872 | vstokes | 172.27.101.234:52628 | NULL | Sleep | 60 | | NULL | | 716902 | vstokes | 172.27.101.234:52660 | NULL | Query | 0 | init | show processlist | +--------+-----------------+----------------------+-------------------+---------+---------+------------------------+-------------------+ 39 rows in set (0.07 sec)
[29 May 2014 12:59]
Van Stokes
We just notice something interesting. This issue appears to only occurs when a MASTER purges its GTIDs and then rotates its binlog. Since we have four masters (circular/loop) then this issue appears to happen quite often.
[29 May 2014 13:26]
lalit Choudhary
Using mysql 5.6.15, MIXED binlog format, all tables with InnoDB engine. my case is same,i have 2 masters(active-active) and each master has a salve. I face this problem once and after that it's working fine for me. I am not sure but different max_allowed_packets size on masters may cause the problem.
[4 Jul 2014 8:45]
Arvind Sharma
Any updates please? This bug is affecting my backup slave almost every night. My environment is as follows: 1. CentOS Linux 6.5 2. MySQL 5.6.15 3. Dedicated slave only for backups using Percona xtrabackup 4. All tables are innodb apart from MySQL specific e.g. mysql schema
[26 Aug 2014 16:12]
Naheem Munir
We have encountered exact same issue, our setup is one master 2 slaves. Running 5.6.18-enterprise-commercial-advanced-log MySQL Enterprise Server - Advanced Edition (Commercial) on VMware using Red Hat Enterprise Linux Server release 6.3 (Santiago) both slaves were restored from Master using MySQL Enterprise Backup. Replication was catching up fine and on both slave on exact same point stopped with the same message Last_Errno:1837 Last_Error: When @@SESSION.GTID_NEXT is set to a GTID, you must explicitly set it to a different value after a COMMIT or ROLLBACK. Please check GTID_NEXT variable manual page for detailed explanation. Current @@SESSION.GTID_NEXT is '6f772a80-0cdb-11e4-9c08-005056837d70:37852877'. workaround: As mentioned in this Bug stop slave; start slave; has restarted the replication back. Please can we have an update thanks
[7 Sep 2015 17:07]
Jon Stephens
Documented fix in the 5.6.27 and 5.7.9 changelogs as follows: When the running with --relay-log-info-repository=TABLE, the slave_relay_log_info table is updated when a transaction is committed or when a flush is performed explicitly, such during relay log rotation. If a transaction that uses nontransactional tables is split across multiple relay logs, it is partially committed on relay log flush. When gtid_mode=ON, this caused the same GTID to be used for the remaining portion of the transaction, which raised an ER_GTID_NEXT_TYPE_UNDEFINED_GROUP error. We fix this issue by postponing in such cases the update of the relay log information repository that normally occurs on relay log rotation until the commit for this transaction has been executed. Closed.