MySQL Bugs: #78627: Segmentation fault after running START GROUP

Bug #78627	Segmentation fault after running START GROUP_REPLICATION on Oracle Linux 6
Submitted:	29 Sep 2015 16:45	Modified:	8 Oct 2015 4:50
Reporter:	Shahriyar Rzayev	Email Updates:
Status:	Not a Bug	Impact on me:	None
Category:	MySQL Server: Group Replication	Severity:	S1 (Critical)
Version:	0.5.0-dmr	OS:	CentOS (6.4)
Assigned to:		CPU Architecture:	Any

Description:
Dear all, decided to test this group replication plugin from articles below:

http://mysqlhighavailability.com/mysql-group-replication-a-small-corosync-guide/
http://mysqlhighavailability.com/getting-started-with-mysql-group-replication/

Downloaded group_replication.so from labs.mysql for Oracle Linux 6 x86_64.

After running:

mysql> START GROUP_REPLICATION;

Got SIGSEGV on gdb side:

Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7fffc8ef2700 (LWP 3294)]
0x00007ffff625d532 in _int_malloc () from /lib64/libc.so.6
Missing separate debuginfos, use: debuginfo-install corosynclib-1.4.7-2.el6.x86_64
(gdb) bt
#0  0x00007ffff625d532 in _int_malloc () from /lib64/libc.so.6
#1  0x00007ffff625e991 in malloc () from /lib64/libc.so.6
#2  0x00007ffff6acf0bd in operator new(unsigned long) () from /usr/lib64/libstdc++.so.6
#3  0x00007fffc8f5b2ab in Certifier::Certifier (this=0x7fffbc051d20)
    at /export/home/pb2/build/sb_0-16452762-1441960761.59/release-0.5.0-export-8447366/BIN/BUILD/mysql-server/plugin/group_replication/src/certifier.cc:209
#4  0x00007fffc8f6bb36 in Certification_handler::initialize (this=0x7fffbc0507e0)
    at /export/home/pb2/build/sb_0-16452762-1441960761.59/release-0.5.0-export-8447366/BIN/BUILD/mysql-server/plugin/group_replication/src/handlers/certification_handler.cc:32
#5  0x00007fffc8f634d7 in configure_pipeline (pipeline=0x7fffbc051a80, handler_list=0x7fffbc0506f0, num_handlers=3)
    at /export/home/pb2/build/sb_0-16452762-1441960761.59/release-0.5.0-export-8447366/BIN/BUILD/mysql-server/plugin/group_replication/src/pipeline_factory.cc:143
#6  0x00007fffc8f6364d in get_pipeline (pipeline_type=<value optimized out>, pipeline=0x7fffbc051a80)
    at /export/home/pb2/build/sb_0-16452762-1441960761.59/release-0.5.0-export-8447366/BIN/BUILD/mysql-server/plugin/group_replication/src/pipeline_factory.cc:30
#7  0x00007fffc8f554b0 in Applier_module::setup_applier_module (this=0x7fffbc051920, pipeline_type=STANDARD_GROUP_REPLICATION_PIPELINE, reset_logs=false, 
    stop_timeout=31536000, group_sidno=2)
    at /export/home/pb2/build/sb_0-16452762-1441960761.59/release-0.5.0-export-8447366/BIN/BUILD/mysql-server/plugin/group_replication/src/applier.cc:99
#8  0x00007fffc8f63eca in configure_and_start_applier_module ()
    at /export/home/pb2/build/sb_0-16452762-1441960761.59/release-0.5.0-export-8447366/BIN/BUILD/mysql-server/plugin/group_replication/src/plugin.cc:761
#9  0x00007fffc8f657df in plugin_group_replication_start ()
    at /export/home/pb2/build/sb_0-16452762-1441960761.59/release-0.5.0-export-8447366/BIN/BUILD/mysql-server/plugin/group_replication/src/plugin.cc:289
#10 0x00000000016927c9 in Group_replication_handler::start (this=0x378f2d0) at /home/full_datadir/mysql-5.7.8-rc/sql/rpl_group_replication.cc:56
#11 0x0000000001692b81 in group_replication_start () at /home/full_datadir/mysql-5.7.8-rc/sql/rpl_group_replication.cc:192
#12 0x000000000149739c in mysql_execute_command (thd=0x7fffbc007180) at /home/full_datadir/mysql-5.7.8-rc/sql/sql_parse.cc:3029
#13 0x000000000149e15b in mysql_parse (thd=0x7fffbc007180, parser_state=0x7fffc8ef14a0) at /home/full_datadir/mysql-5.7.8-rc/sql/sql_parse.cc:5255
#14 0x0000000001492dec in dispatch_command (thd=0x7fffbc007180, com_data=0x7fffc8ef1cd0, command=COM_QUERY)
    at /home/full_datadir/mysql-5.7.8-rc/sql/sql_parse.cc:1272
#15 0x00000000014917f3 in do_command (thd=0x7fffbc007180) at /home/full_datadir/mysql-5.7.8-rc/sql/sql_parse.cc:852
#16 0x00000000015c9042 in handle_connection (arg=0x36c1a50) at /home/full_datadir/mysql-5.7.8-rc/sql/conn_handler/connection_handler_per_thread.cc:300
#17 0x0000000001c56c41 in pfs_spawn_thread (arg=0x37d1990) at /home/full_datadir/mysql-5.7.8-rc/storage/perfschema/pfs.cc:2178
#18 0x00007ffff7bc79d1 in start_thread () from /lib64/libpthread.so.0
#19 0x00007ffff62ccb6d in clone () from /lib64/libc.so.6

Compiled MySQL 5.7.8-rc2 as:

cmake -DCMAKE_INSTALL_PREFIX=/home/full_datadir/mysql-5.7.8-grouprepl -DMYSQL_DATADIR=/home/full_datadir/mysql-5.7.8-grouprepl/datadir -DSYSCONFDIR=/home/full_datadir/mysql-5.7.8-grouprepl -DWITH_SSL=system -DMYSQL_TCP_PORT=3307 -DMYSQL_UNIX_ADDR=/home/full_datadir/mysql-5.7.8-grouprepl/mysqld.sock -DDEFAULT_CHARSET=utf8 -DDEFAULT_COLLATION=utf8_general_ci -DWITH_DEBUG=1 -DCOMPILATION_COMMENT='Group Replication enabled MySQL-5.7.8' -DOPTIMIZER_TRACE=1 -DWITH_ZLIB=system -DWITH_VALGRIND=1 -DCMAKE_C_FLAGS=-DHAVE_purify -DCMAKE_CXX_FLAGS=-DHAVE_purify -DDOWNLOAD_BOOST=1 -DWITH_BOOST=/home/full_datadir/

Started as:

(gdb)run --defaults-file=/home/full_datadir/mysql-5.7.8-grouprepl/my.cnf --basedir=/home/full_datadir/mysql-5.7.8-grouprepl --datadir=/home/full_datadir/mysql --plugin-dir=/home/full_datadir/mysql-5.7.8-grouprepl/lib/plugin --user=root --log-error=/home/full_datadir/mysql-5.7.8-grouprepl/group_replication.err --pid-file=/home/full_datadir/mysql-5.7.8-grouprepl/mysqld-new.pid --socket=/home/full_datadir/mysql-5.7.8-grouprepl/mysqld-new.sock --port=3306 --transaction-write-set-extraction=MURMUR32 --plugin-load=group_replication.so

How to repeat:
1. Download group_replication.so from labs.mysql for Oracle Linux 6 x86_64.
2. Compile MySQL 5.7.8-rc with debug.
3. Do necessary steps from official tutorials.
4. Try to start group replication -> will get segfault

Will upload bt full.

Suggested fix:
No idea.

bt full output

Attachment: bt_full_bug_78627.txt (text/plain), 9.00 KiB.

Hi Shahriyar

Thanks for your interest and for taking the time to test Group Replication.

Upon research all indicates the problem is due to different build parameters for the server and the plugin lib you are using. As they were compiled with different parameters, during runtime problems like this one emerge.

So, first of all, when using the plugin lib, we recommend to use it with the corresponding server version released for OEL6 64 bits.

If not possible than the server and the plugin should be compiled with the same options.

If you still want to compile the server and use the plugin lib, than you should avoid debug compilation options/modes. It should work, but there are no guarantees that even so the plugin will be 100% compatible.

Hi Pedro,

But I am already using OEL6 64 bit for this test. I have downloaded the VM template from: 
http://www.oracle.com/technetwork/server-storage/linux/downloads/vm-for-hol-1896500.html

Also i have downloaded from labs.mysql the corresponding version of plugin.

So you advice to test again with optimized build? ok, i will test it and come back again :)

Hi Shahriyar

Just to be sure, I meant that when using the plugin binary, one of the simplest options would be to use it against the MySQL server released for OL6.

Red Hat Enterprise Linux 6 / Oracle Linux 6 (x86, 64-bit), RPM Bundle / or Server		

at http://dev.mysql.com/downloads/mysql/

As Pedro states, if to compile plugin exactly with the related operating system/architecture  it will succeed.
So in fact i could compile and start Group Replication plugin with 5.7.8-rc-debug + Ubuntu 14.04.

Due to this, will mark this report as a "Not a Bug".
But for further steps, mmaybe it should be mentioned in some documentation, for clarification.