Bug #90488 mysqld crash on "start group_replication" with bootstrap
Submitted: 18 Apr 2018 7:02 Modified: 11 Jul 2018 1:00
Reporter: Peter Hellberg Email Updates:
Status: Can't repeat Impact on me:
None 
Category:MySQL Server: Group Replication Severity:S2 (Serious)
Version:5.7.21 MySQL Communinity Edition OS:Ubuntu (16.04)
Assigned to: Bogdan Kecman CPU Architecture:x86
Tags: bootstrap, crash, group_replication

[18 Apr 2018 7:02] Peter Hellberg
Description:
When attempting to bootstrap the first node for Group Replication, with the following commands:

    SET GLOBAL group_replication_bootstrap_group=ON;
    START GROUP_REPLICATION;
    SET GLOBAL group_replication_bootstrap_group=OFF;

The mysqld process crashes, with the following in the log:

2018-04-17T18:39:50.104575Z 7 [Note] Plugin group_replication reported: 'Group communication SSL configuration: group_replication_ssl_mode: "DISABLED"'
2018-04-17T18:39:50.789544Z 7 [Warning] Plugin group_replication reported: '[GCS] Automatically adding IPv4 localhost address to the whitelist. It is mandatory that it is added.'
2018-04-17T18:39:50.809582Z 7 [Note] Plugin group_replication reported: '[GCS] SSL was not enabled'
2018-04-17T18:39:50.809634Z 7 [Note] Plugin group_replication reported: 'Initialized group communication with configuration: group_replication_group_name: "2441e4ee-eba9-49d2-9d72-8c240f49ae08"; group_replication_local_address: "197.242.158.56:33061"; group_replication_group_seeds: "zadb1.forcelink.net:33061,zadb2.forcelink.net:33061,zadb3.forcelink.net:33061"; group_replication_bootstrap_group: true; group_replication_poll_spin_loops: 0; group_replication_compression_threshold: 1000000; group_replication_ip_whitelist: "197.242.158.56,197.242.157.58,197.242.156.37"'
2018-04-17T18:39:50.809679Z 7 [Note] Plugin group_replication reported: 'Member configuration: member_id: 5; member_uuid: "52f4a1a1-4b99-11e7-80c1-1866da97384c"; single-primary mode: "true"; group_replication_auto_increment_increment: 7; '
2018-04-17T18:39:50.810124Z 12 [Note] 'CHANGE MASTER TO FOR CHANNEL 'group_replication_applier' executed'. Previous state master_host='<NULL>', master_port= 0, master_log_file='', master_log_pos= 4, master_bind=''. New state master_host='<NULL>', master_port= 0, master_log_file='', master_log_pos= 4, master_bind=''.
2018-04-17T18:39:50.847486Z 7 [Note] Plugin group_replication reported: 'Group Replication applier module successfully initialized!'
2018-04-17T18:39:50.847526Z 7 [Note] Plugin group_replication reported: 'auto_increment_increment is set to 7'
2018-04-17T18:39:50.847534Z 7 [Note] Plugin group_replication reported: 'auto_increment_offset is set to 5'
2018-04-17T18:39:50.847534Z 15 [Note] Slave SQL thread for channel 'group_replication_applier' initialized, starting replication in log 'FIRST' at position 0, relay log './zadb-relay-bin-group_replication_applier.000017' position: 20819
2018-04-17T18:39:50.847679Z 0 [Note] Plugin group_replication reported: 'state 0 action xa_init'
2018-04-17T18:39:51.044269Z 0 [Note] Plugin group_replication reported: 'Successfully bound to 0.0.0.0:33061 (socket=165).'
2018-04-17T18:39:51.044301Z 0 [Note] Plugin group_replication reported: 'Successfully set listen backlog to 32 (socket=165)!'
2018-04-17T18:39:51.044308Z 0 [Note] Plugin group_replication reported: 'Successfully unblocked socket (socket=165)!'
2018-04-17T18:39:51.044344Z 0 [Note] Plugin group_replication reported: 'Ready to accept incoming connections on 0.0.0.0:33061 (socket=165)!'
2018-04-17T18:39:51.044382Z 0 [Note] Plugin group_replication reported: 'connecting to 197.242.158.56 33061'
2018-04-17T18:39:51.044504Z 0 [Note] Plugin group_replication reported: 'client connected to 197.242.158.56 33061 fd 166'
2018-04-17T18:39:51.044676Z 0 [Note] Plugin group_replication reported: 'connecting to 197.242.158.56 33061'
*** buffer overflow detected ***: /usr/sbin/mysqld terminated
======= Backtrace: =========
/lib/x86_64-linux-gnu/libc.so.6(+0x777e5)[0x7fa5cbd0c7e5]
/lib/x86_64-linux-gnu/libc.so.6(__fortify_fail+0x5c)[0x7fa5cbdae11c]
/lib/x86_64-linux-gnu/libc.so.6(+0x117120)[0x7fa5cbdac120]
/lib/x86_64-linux-gnu/libc.so.6(+0x119067)[0x7fa5cbdae067]
/usr/lib/mysql/plugin/group_replication.so(+0x67c6a)[0x7fa379b18c6a]
/usr/lib/mysql/plugin/group_replication.so(_ZN19Gcs_xcom_proxy_impl18xcom_open_handlersENSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEt+0xda)[0x7fa379b5fb8a]
/usr/lib/mysql/plugin/group_replication.so(_ZN16Gcs_xcom_control7do_joinEv+0xb41)[0x7fa379b45ac1]
/usr/lib/mysql/plugin/group_replication.so(_ZN26Parameterized_notificationILb0EEclEv+0xa)[0x7fa379b8296a]
/usr/lib/mysql/plugin/group_replication.so(_ZN15Gcs_xcom_engine7processEv+0x76)[0x7fa379b822c6]
/usr/lib/mysql/plugin/group_replication.so(_Z27process_notification_threadPv+0x9)[0x7fa379b82319]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x76ba)[0x7fa5cc9076ba]
/lib/x86_64-linux-gnu/libc.so.6(clone+0x6d)[0x7fa5cbd9c3dd]
======= Memory map: ========
00400000-01a4a000 r-xp 00000000 08:11 21502214                           /usr/sbin/mysqld
01c49000-01d28000 r--p 01649000 08:11 21502214                           /usr/sbin/mysqld
.
.
.
18:39:51 UTC - mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
Attempting to collect some information that could help diagnose the problem.
As this is a crash and something is definitely wrong, the information
collection process might fail.

key_buffer_size=134217728
read_buffer_size=131072
max_used_connections=5
max_threads=300
thread_count=7
connection_count=4
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 250279 K  bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0x0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0 thread_stack 0x30000
/usr/sbin/mysqld(my_print_stacktrace+0x3b)[0xe8ffab]
/usr/sbin/mysqld(handle_fatal_signal+0x489)[0x787f89]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x11390)[0x7fa5cc911390]
/lib/x86_64-linux-gnu/libc.so.6(gsignal+0x38)[0x7fa5cbcca428]
/lib/x86_64-linux-gnu/libc.so.6(abort+0x16a)[0x7fa5cbccc02a]
/lib/x86_64-linux-gnu/libc.so.6(+0x777ea)[0x7fa5cbd0c7ea]
/lib/x86_64-linux-gnu/libc.so.6(__fortify_fail+0x5c)[0x7fa5cbdae11c]
/lib/x86_64-linux-gnu/libc.so.6(+0x117120)[0x7fa5cbdac120]
/lib/x86_64-linux-gnu/libc.so.6(+0x119067)[0x7fa5cbdae067]
/usr/lib/mysql/plugin/group_replication.so(+0x67c6a)[0x7fa379b18c6a]
/usr/lib/mysql/plugin/group_replication.so(_ZN19Gcs_xcom_proxy_impl18xcom_open_handlersENSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEt+0xda)[0x7fa379b5fb8a]
/usr/lib/mysql/plugin/group_replication.so(_ZN16Gcs_xcom_control7do_joinEv+0xb41)[0x7fa379b45ac1]
/usr/lib/mysql/plugin/group_replication.so(_ZN26Parameterized_notificationILb0EEclEv+0xa)[0x7fa379b8296a]
/usr/lib/mysql/plugin/group_replication.so(_ZN15Gcs_xcom_engine7processEv+0x76)[0x7fa379b822c6]
/usr/lib/mysql/plugin/group_replication.so(_Z27process_notification_threadPv+0x9)[0x7fa379b82319]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x76ba)[0x7fa5cc9076ba]
/lib/x86_64-linux-gnu/libc.so.6(clone+0x6d)[0x7fa5cbd9c3dd]
The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you find out what is causing the crash.

How to repeat:
My config settings:

[mysqld]
open_files_limit = 65535
user            = mysql
pid-file        = /var/run/mysqld/mysqld.pid
socket          = /var/run/mysqld/mysqld.sock
port            = 3306
basedir         = /usr
datadir         = /var/lib/mysql
tmpdir          = /tmp
lc-messages-dir = /usr/share/mysql
skip-external-locking
skip-name-resolve
lower_case_table_names=1
group_concat_max_len = 4096
net-write-timeout=1200
net-read-timeout=1200
wait_timeout=1200
log-bin=mysql-bin
log-bin-trust-function-creators=true
max_connect_errors=10000
relay_log=zadb-relay-bin
innodb_print_all_deadlocks = 1
innodb_buffer_pool_size=8G
innodb_read_io_threads=8
innodb_write_io_threads=8
optimizer_search_depth=62
innodb_stats_on_metadata=0
innodb_log_file_size = 536870912
bind-address            = 0.0.0.0
key_buffer_size         = 128M
max_allowed_packet      = 64M
thread_stack            = 192K
thread_cache_size       = 8
myisam_recover_options  = BACKUP
max_connections        = 300
query_cache_limit       = 4M
query_cache_size        = 10M
log_error = /var/log/mysql/error.log
slow_query_log=1
long_query_time=2
slow_query_log_file        = /var/log/mysql/mysql-slow.log
expire_logs_days        = 10
max_binlog_size         = 100M

# General replication settings
gtid_mode = ON
enforce_gtid_consistency = ON
master_info_repository = TABLE
relay_log_info_repository = TABLE
binlog_checksum = NONE
log_slave_updates = ON
log_bin = binlog
binlog_format = ROW
transaction_write_set_extraction = XXHASH64
loose-group_replication_bootstrap_group = OFF
loose-group_replication_start_on_boot = OFF
loose-group_replication_ssl_mode = DISABLED
loose-group_replication_recovery_use_ssl = 0

# Shared replication group configuration
loose-group_replication_group_name = "2441e4ee-eba9-49d2-9d72-8c240f49ae08"
loose-group_replication_ip_whitelist = "197.242.158.56,197.242.157.58,197.242.156.37"
loose-group_replication_group_seeds = "zadb1.forcelink.net:33061,zadb2.forcelink.net:33061,zadb3.forcelink.net:33061"

# Host specific replication configuration
server_id = 5
report_host = "zadb1.forcelink.net"
loose-group_replication_local_address = "197.242.158.56:33061"
[11 Jul 2018 1:00] Bogdan Kecman
Hi Peter,

I'm sorry but I can't reproduce this issue. Tried with your config, tried with some changes too and I just can't reproduce the problem?

Are you still seeing this problem?

all best
Bogdan