Bug #96915 mysql crash
Submitted: 18 Sep 3:33 Modified: 7 Oct 14:46
Reporter: HANWEN YIN Email Updates:
Status: Duplicate Impact on me:
Category:MySQL Server Severity:S2 (Serious)
Version:8.0.15 OS:Linux
Assigned to: Bogdan Kecman CPU Architecture:x86
Tags: mysql crash mgr

[18 Sep 3:33] HANWEN YIN
mysql  Ver 8.0.15 for linux-glibc2.12 on x86_64 (MySQL Community Server - GPL)

Linux mysql-prd 3.10.0-693.2.2.el7.x86_64 #1 SMP Tue Sep 12 22:26:13 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux

mysql binary:mysql-8.0.15-linux-glibc2.12-x86_64.tar.xz

mgr with 3 node,single primary mode.crash node is secondary node.
can not reproduce.

$free -m
              total        used        free      shared  buff/cache   available
Mem:          32011        6249        6122           1       19639       24218
Swap:             0           0           0

mysql crash info:
15:44:06 UTC - mysqld got signal 11 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
Attempting to collect some information that could help diagnose the problem.
As this is a crash and something is definitely wrong, the information
collection process might fail.

It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 2390240 K  bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0x2b8ea001c530
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 2b8cabfb7d80 thread_stack 0x40000
/ds/mysql/base/8.0/bin/mysqld(my_print_stacktrace(unsigned char*, unsigned long)+0x2e) [0x1c6d98e]
/ds/mysql/base/8.0/bin/mysqld(handle_fatal_signal+0x413) [0xe261e3]
/lib64/libpthread.so.0(+0xf5e0) [0x2b8c789385e0]
/lib64/libc.so.6(+0x14c5d6) [0x2b8c7a5ac5d6]
/lib64/libstdc++.so.6(std::string::_Rep::_M_clone(std::allocator<char> const&, unsigned long)+0x40) [0x2b8c79cfe930]
/lib64/libstdc++.so.6(std::basic_string<char, std::char_traits<char>, std::allocator<char> >::basic_string(std::string const&)+0x5c) [0x2b8c79cfefdc]
/ds/mysql/base/8.0/lib/plugin/group_replication.so(Flow_control_module::get_pipeline_stats(std::string const&)+0x99) [0x2b8e5e11ec49]
/ds/mysql/base/8.0/lib/plugin/group_replication.so(get_group_member_stats(unsigned int, GROUP_REPLICATION_GROUP_MEMBER_STATS_CALLBACKS const&, Group_member_info_manager_interface*, Applier_module*, Gcs_operations*, char*)+0x4e2) [0x2b8e5e12ea72]
/ds/mysql/base/8.0/bin/mysqld(get_group_replication_group_member_stats_info(unsigned int, GROUP_REPLICATION_GROUP_MEMBER_STATS_CALLBACKS const&)+0x3f) [0x108a42f]
/ds/mysql/base/8.0/bin/mysqld(table_replication_group_member_stats::make_row(unsigned int)+0x137) [0x1df6267]
/ds/mysql/base/8.0/bin/mysqld(table_replication_group_member_stats::rnd_next()+0x39) [0x1df62a9]
/ds/mysql/base/8.0/bin/mysqld(ha_perfschema::rnd_next(unsigned char*)+0x4e) [0x1d7907e]
/ds/mysql/base/8.0/bin/mysqld(handler::ha_rnd_next(unsigned char*)+0x18c) [0xf178cc]
/ds/mysql/base/8.0/bin/mysqld(TableScanIterator::Read()+0x1d) [0xc7df6d]
/ds/mysql/base/8.0/bin/mysqld(sub_select(JOIN*, QEP_TAB*, bool)+0x279) [0xce3f19]
/ds/mysql/base/8.0/bin/mysqld(JOIN::exec()+0x421) [0xce0821]
/ds/mysql/base/8.0/bin/mysqld(Sql_cmd_dml::execute_inner(THD*)+0x104) [0xd5a714]
/ds/mysql/base/8.0/bin/mysqld(Sql_cmd_dml::execute(THD*)+0x2d1) [0xd62351]
/ds/mysql/base/8.0/bin/mysqld(mysql_execute_command(THD*, bool)+0x27cb) [0xd145eb]
/ds/mysql/base/8.0/bin/mysqld(mysql_parse(THD*, Parser_state*, bool)+0x33f) [0xd1657f]
/ds/mysql/base/8.0/bin/mysqld(dispatch_command(THD*, COM_DATA const*, enum_server_command)+0x2e1b) [0xd1980b]
/ds/mysql/base/8.0/bin/mysqld(do_command(THD*)+0x179) [0xd1a2c9]
/ds/mysql/base/8.0/bin/mysqld() [0xe184b8]
/ds/mysql/base/8.0/bin/mysqld() [0x1d7d6ff]
/lib64/libpthread.so.0(+0x7e25) [0x2b8c78930e25]
/lib64/libc.so.6(clone+0x6d) [0x2b8c7a55834d]

Trying to get some variables.
Some pointers may be invalid and cause the dump to abort.
Query (2b8ea00117e8): SELECT viable_candidate,read_only,transactions_behind FROM sys.gr_member_routing_candidate_status
Connection ID (thread ID): 331422

The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you find out what is causing the crash.

How to repeat:
can not reproduce.
[18 Sep 10:11] Bogdan Kecman

This is not enough data for us to do anything :(

Can you please
 - provide config from all nodes
 - tell us if you know what were you doing when the crash happens
 - check log files on all nodes

kind regards
[26 Sep 2:12] Francis Lavalliere
Hello we are using docker mysql with 8.0.17  3 node group replication in multi-master mode.

We have tested various items, including the use of jemalloc i believe there could be a memory leak somewhere maybe? Could be unrelated, but in our environment it looks like memory simply keep growing until we do a supervisor restat mysql type of thing.

Might be totally unrelated.
[26 Sep 3:17] Francis Lavalliere
I found out in my case (and maybe in this case too). that event_scheduler was enabled. There is another bug with event_scheduler causing memory leak in mysql 8.
[26 Sep 13:50] Bogdan Kecman
Yes, This could be duplicate of bug #95787 but there's just not enough info to confirm or deny that.
[26 Sep 14:32] Francis Lavalliere
I'll try to add informations here since i disabled event_scheduler in my systems.

Slaves after 12 hours memory is at : 771MiB
Master is at 1.012GiB

I also have performance schema disabled / metrics (as it was not matching the total memory usage).

Let me know if there is something I can provide (core dump or anything/ what commands to run) that could be of any help? maybe.
[7 Oct 14:46] Bogdan Kecman
Thanks for your data Francis, looking at this it looks like this is a duplicate of bug #95787