Bug #103316 Group replication assertion / memory corruption
Submitted: 14 Apr 2021 11:49 Modified: 14 Apr 2021 15:55
Reporter: Marcelo Altmann (OCA) Email Updates:
Status: Verified Impact on me:
None 
Category:MySQL Server: Group Replication Severity:S6 (Debug Builds)
Version:8.0.23 OS:Any
Assigned to: CPU Architecture:Any

[14 Apr 2021 11:49] Marcelo Altmann
Description:
Group Replication is asserting when bootstraping the cluster on debug builds:

11:48:51 UTC - mysqld got signal 11 ;
*** Error in `/tmp/mysql-8.0.23-linux-glibc2.12-x86_64/bin/mysqld-debug': malloc(): memory corruption: 0x00007fbb78168490 ***
======= Backtrace: =========
/lib64/libc.so.6(+0x82aa6)[0x7fbc2e423aa6]
/lib64/libc.so.6(__libc_malloc+0x4c)[0x7fbc2e4266fc]
/lib64/libstdc++.so.6(_Znwm+0x1d)[0x7fbc2ece618d]
/tmp/mysql-8.0.23-linux-glibc2.12-x86_64/lib/plugin/group_replication.so(_ZN9CertifierC1Ev+0x36c)[0x7fbbe0458abc]
/tmp/mysql-8.0.23-linux-glibc2.12-x86_64/lib/plugin/group_replication.so(_ZN21Certification_handler10initializeEv+0x1f)[0x7fbbe0480d9f]
/tmp/mysql-8.0.23-linux-glibc2.12-x86_64/lib/plugin/group_replication.so(_Z18configure_pipelinePP13Event_handlerP10Handler_idi+0x7a)[0x7fbbe04a7faa]
/tmp/mysql-8.0.23-linux-glibc2.12-x86_64/lib/plugin/group_replication.so(_Z12get_pipeline21Handler_pipeline_typePP13Event_handler+0x2f)[0x7fbbe04a87df]
/tmp/mysql-8.0.23-linux-glibc2.12-x86_64/lib/plugin/group_replication.so(_ZN14Applier_module20setup_applier_moduleE21Handler_pipeline_typebmiyP16Shared_writelock+0x1a1)[0x7fbbe044b5e1]
/tmp/mysql-8.0.23-linux-glibc2.12-x86_64/lib/plugin/group_replication.so(_Z34configure_and_start_applier_modulev+0xc7)[0x7fbbe04b0b37]
/tmp/mysql-8.0.23-linux-glibc2.12-x86_64/lib/plugin/group_replication.so(_Z25initialize_plugin_modulesSt6bitsetILm18EE+0x125)[0x7fbbe04b1675]
/tmp/mysql-8.0.23-linux-glibc2.12-x86_64/lib/plugin/group_replication.so(_Z26initialize_plugin_and_join25enum_plugin_con_isolationP29Delayed_initialization_thread+0x14a)[0x7fbbe04b7a2a]
/tmp/mysql-8.0.23-linux-glibc2.12-x86_64/lib/plugin/group_replication.so(_Z30plugin_group_replication_startPPc+0x146b)[0x7fbbe04b96db]
/tmp/mysql-8.0.23-linux-glibc2.12-x86_64/bin/mysqld-debug(_Z23group_replication_startPPcP3THD+0x200)[0x3759d15]
/tmp/mysql-8.0.23-linux-glibc2.12-x86_64/bin/mysqld-debug(_Z21mysql_execute_commandP3THDb+0x17a1)[0x3137311]
/tmp/mysql-8.0.23-linux-glibc2.12-x86_64/bin/mysqld-debug(_Z20dispatch_sql_commandP3THDP12Parser_state+0x6b7)[0x313cc20]
/tmp/mysql-8.0.23-linux-glibc2.12-x86_64/bin/mysqld-debug(_Z16dispatch_commandP3THDPK8COM_DATA19enum_server_command+0x14dd)[0x313328e]
/tmp/mysql-8.0.23-linux-glibc2.12-x86_64/bin/mysqld-debug(_Z10do_commandP3THD+0x482)[0x31316de]
/tmp/mysql-8.0.23-linux-glibc2.12-x86_64/bin/mysqld-debug[0x3300855]
/tmp/mysql-8.0.23-linux-glibc2.12-x86_64/bin/mysqld-debug[0x4edc0d5]
/lib64/libpthread.so.0(+0x7ea5)[0x7fbc303baea5]
/lib64/libc.so.6(clone+0x6d)[0x7fbc2e49f96d]

How to repeat:
- wget https://dev.mysql.com/get/Downloads/MySQL-8.0/mysql-8.0.23-linux-glibc2.12-x86_64.tar.xz
- tar -xvf mysql-8.0.23-linux-glibc2.12-x86_64.tar.xz
- cd /tmp/mysql-8.0.23-linux-glibc2.12-x86_64
- bin/mysqld-debug --datadir=data --initialize-insecure
- bin/mysqld-debug --user=mysql --datadir=data --basedir=/tmp/mysql-8.0.23-linux-glibc2.12-x86_64 --plugin_load_add='group_replication.so' --loose-group_replication_group_name="aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa" --loose-group_replication_start_on_boot=off --loose-group_replication_bootstrap_group=off --loose-group_replication_recovery_use_ssl=ON --loose-group_replication_recovery_get_public_key=ON --loose-group_replication_local_address="127.0.0.1:13826" --gtid_mode=ON --enforce-gtid-consistency &
- bin/mysql -e "SET SQL_LOG_BIN=0;
    CREATE USER rpl_user@'localhost' IDENTIFIED BY 'password' REQUIRE SSL;
    GRANT REPLICATION SLAVE ON *.* TO rpl_user@'localhost';
    GRANT BACKUP_ADMIN ON *.* TO rpl_user@'localhost';
    FLUSH PRIVILEGES;
    SET SQL_LOG_BIN=1;
    CHANGE MASTER TO MASTER_USER='rpl_user', MASTER_PASSWORD='password' FOR CHANNEL 'group_replication_recovery';
SET GLOBAL group_replication_bootstrap_group=ON;
        START GROUP_REPLICATION USER='rpl_user', PASSWORD='password';"
[14 Apr 2021 12:24] MySQL Verification Team
Hello Marcelo,

Thank you for the report and feedback.
Observed that debug build is affected.

Thanks,
Umesh
[14 Apr 2021 12:33] MySQL Verification Team
MySQL Server 8.0.23 test results

Attachment: 103316_8.0.23.results (application/octet-stream, text), 24.98 KiB.

[14 Apr 2021 15:55] Marcelo Altmann
I cannot reproduce by just compiling it with debug, however (and I believe that is how tarball is made), if I compile it with Debug, save mysqld as mysqld-debug, then compile it again as RelWithDebInfo and execute the previous mysqld-debug it crashes the same way. 

So the issue is mysqld using Debug while group_replication.so is on RelWithDebInfo