Bug #98082 MySQL Community Server 8.0.18 got signal 6
Submitted: 27 Dec 2019 8:04 Modified: 6 Mar 2020 18:49
Reporter: yue can Email Updates:
Status: Duplicate Impact on me:
None 
Category:MySQL Server Severity:S1 (Critical)
Version:8.0.18 OS:CentOS (7.5)
Assigned to: MySQL Verification Team CPU Architecture:Any
Tags: mgr, MySQL8.0.18, signal 6

[27 Dec 2019 8:04] yue can
Description:
Environment:MySQL Community Server 8.0.18,MGR singel primary,3 nodes
ERROR:The MySQL service suddenly crashes. The error log prompts "MySQL got signal 6"
RESULT:mgr member failed and mysql service suddenly crashes 
the error log  is as follows:
2019-12-27T04:45:03.053496+08:00 548963 [Note] [MY-013417] [Server] The wait_timeout period was exceeded, the idle time since last command was too long.
2019-12-27T04:45:03.053547+08:00 548963 [Note] [MY-010914] [Server] Aborted connection 548963 to db: 'unconnected' user: 'xx' host: '192.168.xx.xx' (Got timeout reading communication packets).
terminate called after throwing an instance of 'std::bad_alloc'
  what():  std::bad_alloc
20:51:32 UTC - mysqld got signal 6 ;
Most likely, you have hit a bug, but this error can also be caused by malfunctioning hardware.
Thread pointer: 0x7fb3a80928e0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 7fb49738dd80 thread_stack 0x80000
/usr/local/mysql8018/bin/mysqld(my_print_stacktrace(unsigned char const*, unsigned long)+0x2e) [0x1ed6cce]
/usr/local/mysql8018/bin/mysqld(handle_fatal_signal+0x323) [0xfb2d23]
/lib64/libpthread.so.0(+0xf6d0) [0x7fb91f5406d0]
/lib64/libc.so.6(gsignal+0x37) [0x7fb91d5ff277]
/lib64/libc.so.6(abort+0x148) [0x7fb91d600968]
/lib64/libstdc++.so.6(__gnu_cxx::__verbose_terminate_handler()+0x165) [0x7fb91df0eac5]
/lib64/libstdc++.so.6(+0x5ea36) [0x7fb91df0ca36]
/lib64/libstdc++.so.6(+0x5ea63) [0x7fb91df0ca63]
/lib64/libstdc++.so.6(+0x5ec83) [0x7fb91df0cc83]
/lib64/libstdc++.so.6(operator new(unsigned long)+0x7d) [0x7fb91df0d21d]
/lib64/libstdc++.so.6(std::string::_Rep::_S_create(unsigned long, unsigned long, std::allocator<char> const&)+0x59) [0x7fb91df6bda9]
/lib64/libstdc++.so.6(std::string::_Rep::_M_clone(std::allocator<char> const&, unsigned long)+0x1b) [0x7fb91df6c9bb]
/lib64/libstdc++.so.6(std::basic_string<char, std::char_traits<char>, std::allocator<char> >::basic_string(std::string const&)+0x5c) [0x7fb91df6d08c]
/usr/local/mysql-8.0.18-linux-glibc2.12-x86_64/lib/plugin/group_replication.so(Flow_control_module::get_pipeline_stats(std::string const&)+0x99) [0x7fb60395f799]
/usr/local/mysql-8.0.18-linux-glibc2.12-x86_64/lib/plugin/group_replication.so(get_group_member_stats(unsigned int, GROUP_REPLICATION_GROUP_MEMBER_STATS_CALLBACKS const&, Group_member_info_manager_interface*, Applier_module*, Gcs_operations*, char*)+0x696) [0x7fb603985556]
/usr/local/mysql8018/bin/mysqld(get_group_replication_group_member_stats_info(unsigned int, GROUP_REPLICATION_GROUP_MEMBER_STATS_CALLBACKS const&)+0x43) [0x1264ab3]
/usr/local/mysql8018/bin/mysqld(table_replication_group_member_stats::make_row(unsigned int)+0x137) [0x242a7d7]
/usr/local/mysql8018/bin/mysqld(table_replication_group_member_stats::rnd_next()+0x46) [0x242a826]
/usr/local/mysql8018/bin/mysqld(ha_perfschema::rnd_next(unsigned char*)+0x4e) [0x23ace2e]
/usr/local/mysql8018/bin/mysqld(handler::ha_rnd_next(unsigned char*)+0x194) [0x10bcc34]
/usr/local/mysql8018/bin/mysqld(TableScanIterator::Read()+0x1d) [0xdda36d]
/usr/local/mysql8018/bin/mysqld(HashJoinIterator::ReadRowFromProbeIterator()+0x14) [0x10cc744]
/usr/local/mysql8018/bin/mysqld(HashJoinIterator::Read()+0xc2) [0x10cc962]
/usr/local/mysql8018/bin/mysqld(FilterIterator::Read()+0x14) [0x106cfb4]
/usr/local/mysql8018/bin/mysqld(SELECT_LEX_UNIT::ExecuteIteratorQuery(THD*)+0x2cb) [0xf412cb]
/usr/local/mysql8018/bin/mysqld(SELECT_LEX_UNIT::execute(THD*)+0xf9) [0xf43489]
/usr/local/mysql8018/bin/mysqld(subselect_iterator_engine::exec(THD*)+0x1a) [0x11a389a]
/usr/local/mysql8018/bin/mysqld(Item_subselect::exec(THD*)+0x185) [0x11a4f65]
/usr/local/mysql8018/bin/mysqld(Item_singlerow_subselect::val_str(String*)+0x4e) [0x11a3afe]
/usr/local/mysql8018/bin/mysqld(Item::save_in_field_inner(Field*, bool)+0xfa) [0x10e9a8a]
/usr/local/mysql8018/bin/mysqld(Item::save_in_field(Field*, bool)+0x44) [0x11017b4]
/usr/local/mysql8018/bin/mysqld(sp_eval_expr(THD*, Field*, Item**)+0xa2) [0x126b412]
/usr/local/mysql8018/bin/mysqld(sp_lex_instr::reset_lex_and_exec_core(THD*, unsigned int*, bool)+0x5ce) [0xe0209e]
/usr/local/mysql8018/bin/mysqld(sp_lex_instr::validate_lex_and_execute_core(THD*, unsigned int*, bool)+0xa2) [0xe02802]
/usr/local/mysql8018/bin/mysqld(sp_head::execute(THD*, bool)+0x58e) [0xdf9bde]
/usr/local/mysql8018/bin/mysqld(sp_head::execute_function(THD*, Item**, unsigned int, Field*)+0x41a) [0xdfbcfa]
/usr/local/mysql8018/bin/mysqld(Item_func_sp::execute_impl(THD*)+0x11d) [0x116775d]
/usr/local/mysql8018/bin/mysqld(Item_func_sp::execute()+0x76) [0x11678e6]
/usr/local/mysql8018/bin/mysqld(Item_func_sp::val_str(String*)+0x39) [0x1167a89]
/usr/local/mysql8018/bin/mysqld(Item::send(Protocol*, String*)+0x205) [0x10e57f5]
/usr/local/mysql8018/bin/mysqld(THD::send_result_set_row(List<Item>*)+0xbb) [0xe21f2b]
/usr/local/mysql8018/bin/mysqld(Query_result_send::send_data(THD*, List<Item>&)+0x31) [0x125ca91]
/usr/local/mysql8018/bin/mysqld(SELECT_LEX_UNIT::ExecuteIteratorQuery(THD*)+0x2aa) [0xf412aa]
/usr/local/mysql8018/bin/mysqld(SELECT_LEX_UNIT::execute(THD*)+0xf9) [0xf43489]
/usr/local/mysql8018/bin/mysqld(Sql_cmd_dml::execute_inner(THD*)+0x20b) [0xed0d4b]
/usr/local/mysql8018/bin/mysqld(Sql_cmd_dml::execute(THD*)+0x3e8) [0xed98b8]
/usr/local/mysql8018/bin/mysqld(mysql_execute_command(THD*, bool)+0x3af7) [0xe8cb77]
/usr/local/mysql8018/bin/mysqld(mysql_parse(THD*, Parser_state*)+0x327) [0xe8e8b7]
/usr/local/mysql8018/bin/mysqld(dispatch_command(THD*, COM_DATA const*, enum_server_command)+0x2474) [0xe91244]
/usr/local/mysql8018/bin/mysqld(do_command(THD*)+0x174) [0xe91cb4]
/usr/local/mysql8018/bin/mysqld() [0xfa4c78]
/usr/local/mysql8018/bin/mysqld() [0x23b04ac]
/lib64/libpthread.so.0(+0x7e25) [0x7fb91f538e25]
/lib64/libc.so.6(clone+0x6d) [0x7fb91d6c7bad]

Trying to get some variables.
Some pointers may be invalid and cause the dump to abort.
Query (7fb3a8012b88): is an invalid pointer
Connection ID (thread ID): 539640
Status: NOT_KILLED

The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you find out what is causing the crash.

How to repeat:
Unknown
Linux server has enough memory and CPU resources.It happened all of a sudden,
[27 Dec 2019 14:14] MySQL Verification Team
Thank you for the bug report. We need a repeatable test case, when you will able to provide it, please add here.
[4 Feb 2020 23:03] Andrew Ernst
sar output for memory

Attachment: mysql-crash-sar.txt (text/plain), 3.02 KiB.

[4 Feb 2020 23:03] Andrew Ernst
I've seen this issue repeat approximately 4 times across different hosts in different environments (all Azure IaaS) when the hosts were only approximately 85% of memory utilized RSS with 8.0.18 (we've been running with `innodb_dedicated_server=1` but as a precaution are in the process of dropping our `innodb_buffer_pool_size` to 65% of total RAM to potentially avoid hitting this bug.

We did not see this behavior prior to 8.0.18.  All our hosts are using Group Replication and these events happened on members carrying the PRIMARY role.

See crash log (mysqld.log) and sar -r output (1 minute granularity)
[6 Mar 2020 18:49] MySQL Verification Team
duplicate of internal Bug 30049349 , fixed in 8.0.20