Bug #101137 Assertion `lost_gtids->is_empty()' failed when mysqld started
Submitted: 12 Oct 2020 17:03 Modified: 5 Mar 2021 13:36
Reporter: Cheng Zhou Email Updates:
Status: Closed Impact on me:
None 
Category:MySQL Server: Logging Severity:S6 (Debug Builds)
Version:8.0 OS:Any
Assigned to: CPU Architecture:Any

[12 Oct 2020 17:03] Cheng Zhou
Description:
When binary log files are automatically removed after their expiration period ends at the startup of mysqld, and the oldest unremoved binary log file has a set of previous-GTIDs, an assertion 'DBUG_ASSERT(lost_gtids->is_empty())' failed:

mysqld: /home/db8021/setup/mysql-8.0.21/sql/mysqld.cc:7065: int mysqld_main(int, char**): Assertion `lost_gtids->is_empty()' failed.
15:53:29 UTC - mysqld got signal 6 ;
Most likely, you have hit a bug, but this error can also be caused by malfunctioning hardware.
Thread pointer: 0x0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0 thread_stack 0x46000
/home/db8021/bin/mysqld(my_print_stacktrace(unsigned char const*, unsigned long)+0x43) [0x4d9efe4]
/home/db8021/bin/mysqld(handle_fatal_signal+0x2bf) [0x3a16406]
/usr/lib64/libpthread.so.0(+0xf5e0) [0x7f8298eee5e0]
/usr/lib64/libc.so.6(gsignal+0x37) [0x7f829709c1f7]
/usr/lib64/libc.so.6(abort+0x148) [0x7f829709d8e8]
/usr/lib64/libc.so.6(+0x2e266) [0x7f8297095266]
/usr/lib64/libc.so.6(+0x2e312) [0x7f8297095312]
/home/db8021/bin/mysqld(mysqld_main(int, char**)+0x1d3f) [0x36963f8]
/home/db8021/bin/mysqld(main+0x20) [0x36839d6]
/usr/lib64/libc.so.6(__libc_start_main+0xf5) [0x7f8297088c05]
/home/db8021/bin/mysqld() [0x36838e9]

How to repeat:
1、mysqld is DEBUG version(compiled with a cmake option: -DWITH_DEBUG=1).
2、In configuration file my.cnf, add:
log_bin = on
gtid_mode = on
enforce_gtid_consistency = on
binlog_expire_logs_seconds = 120 (a small value, so we can repeat quickly)

3、
(1) Execute "mysql.server start" to start mysqld. 
(2) Run some SQLs to make sure that some gitds were generated, for example: create database db1;.
(3) Execute 'flush logs' to rotate the binary log file. Make sure the new binlog file has Previous-GTIDs.
(4) Wait for more than binlog_expire_logs_seconds.
(5) Execute shell cmd "mysql.server restart", then the assertion `lost_gtids->is_empty()' would failed 

Suggested fix:
mysqld.cc line 7006:    Gtid_set *lost_gtids = const_cast<Gtid_set *>(gtid_state->get_lost_gtids());
After this line, lost_gtids already has the gtid-set the same as Previous-GTIDs in the oldest unremoved binlog file in this situation.

1. Just remove the DBUG_ASSERT in mysqld.cc 7065:     DBUG_ASSERT(lost_gtids->is_empty());
2. mysqld.cc 7067 and 7068: 
lost_gtids->add_gtid_set(&purged_gtids_from_binlog) != RETURN_STATUS_OK

Actually, before the IF condition, lost_gtids already includes purged_gtids_from_binlog. The second IF condition "lost_gtids->add_gtid_set(&purged_gtids_from_binlog) != RETURN_STATUS_OK" should also be removed.
[13 Oct 2020 12:23] MySQL Verification Team
Hi Mr. Zhou,

Thank you for your bug report.

I have managed to repeat the behaviour.

Verified as reported, but only for debug builds.
[13 Oct 2020 12:24] MySQL Verification Team
Setting the correct version.
[5 Mar 2021 13:36] Margaret Fisher
Posted by developer:
 
Changelog entry added for MySQL Shell 8.0.24:

An assertion was raised in debug builds relating to lost GTIDs if binary log files were removed at startup because their retention period had expired.
[5 Mar 2021 13:37] MySQL Verification Team
Thank you Mrs. Fisher.