Bug #83013 Segfault related to semisync master
Submitted: 15 Sep 2016 12:36 Modified: 27 Sep 2016 5:55
Reporter: Daniël van Eeden (OCA) Email Updates:
Status: Duplicate Impact on me:
None 
Category:MySQL Server: Replication Severity:S2 (Serious)
Version:5.7.14 OS:CentOS (6.8)
Assigned to: CPU Architecture:Any
Tags: crash, semisync

[15 Sep 2016 12:36] Daniël van Eeden
Description:
2016-09-15T07:02:02.143947Z 3634412 [Warning] Timeout waiting for reply of binlog (file: binlog.003260, pos: 23854587), semi-sync up to file binlog.003260, position 23854250.
2016-09-15T07:30:21.669921Z 3642636 [Warning] Timeout waiting for reply of binlog (file: binlog.003260, pos: 24548583), semi-sync up to file binlog.003260, position 24548246.
09:10:43 UTC - mysqld got signal 11 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
Attempting to collect some information that could help diagnose the problem.
As this is a crash and something is definitely wrong, the information
collection process might fail.

key_buffer_size=805306368
read_buffer_size=131072
max_used_connections=13
max_threads=3000
thread_count=6
connection_count=5
It is possible that mysqld could use up to 
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 4282463 K  bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0x0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0 thread_stack 0x40000
/usr/sbin/mysqld(my_print_stacktrace+0x35)[0xf3e845]
/usr/sbin/mysqld(handle_fatal_signal+0x4a4)[0x7cdb24]
/lib64/libpthread.so.0(+0xf7e0)[0x7fc0d46bd7e0]
/usr/sbin/mysqld[0xc628a2]
/usr/sbin/mysqld(my_net_read+0x18f)[0xc62b7f]
/usr/lib64/mysql/plugin/semisync_master.so(_ZN12Ack_receiver3runEv+0x3ec)[0x7fb9b06acaec]
/usr/lib64/mysql/plugin/semisync_master.so(ack_receive_handler+0x19)[0x7fb9b06acdc9]
/usr/sbin/mysqld(pfs_spawn_thread+0x174)[0xf5cd44]
/lib64/libpthread.so.0(+0x7aa1)[0x7fc0d46b5aa1]
/lib64/libc.so.6(clone+0x6d)[0x7fc0d3213aad]
The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you find out what is causing the crash.
2016-09-15T09:10:43.625712Z mysqld_safe Number of processes running now: 0
2016-09-15T09:10:43.627521Z mysqld_safe mysqld restarted

Demangled stack trace:

/usr/sbin/mysqld(my_print_stacktrace+0x35)[0xf3e845]
/usr/sbin/mysqld(handle_fatal_signal+0x4a4)[0x7cdb24]
/lib64/libpthread.so.0(+0xf7e0)[0x7fc0d46bd7e0]
/usr/sbin/mysqld[0xc628a2]
/usr/sbin/mysqld(my_net_read+0x18f)[0xc62b7f]
/usr/lib64/mysql/plugin/semisync_master.so(Ack_receiver::run()+0x3ec)[0x7fb9b06acaec]
/usr/lib64/mysql/plugin/semisync_master.so(ack_receive_handler+0x19)[0x7fb9b06acdc9]
/usr/sbin/mysqld(pfs_spawn_thread+0x174)[0xf5cd44]
/lib64/libpthread.so.0(+0x7aa1)[0x7fc0d46b5aa1]
/lib64/libc.so.6(clone+0x6d)[0x7fc0d3213aad]

From the my.cnf:
plugin-load-add='rpl_semi_sync_master=semisync_master.so'
rpl_semi_sync_master_enabled=1
rpl_semi_sync_master_wait_no_slave=0
plugin-load-add='rpl_semi_sync_slave=semisync_slave.so'
rpl_semi_sync_slave_enabled=1

How to repeat:
Unknown
[16 Sep 2016 5:55] MySQL Verification Team
Hello Daniël,

Thank you for the report.
I'm unable to locate any closest matching bug for this issue, and also not able to reproduce with simple setup and with sysbench load.  
I know you usually provide repeatable test cases but still would like to ask, do you have a repeatable test case for this? Please let us know.

Thanks,
Umesh
[16 Sep 2016 6:21] Daniël van Eeden
Sorry, I don't have a repeatable testcase or any info on what might have triggered this.
[16 Sep 2016 7:42] Daniël van Eeden
Might be related to:
Bug #79865 	semi-sync can't run normally when many connections
[27 Sep 2016 5:55] Daniël van Eeden
Duplicate of Bug #79865