Bug #97423 MySQL 8.0 crashes under low memory
Submitted: 30 Oct 2019 10:10 Modified: 30 Nov 2019 10:43
Reporter: Dmitry Smal Email Updates:
Status: No Feedback Impact on me:
None 
Category:MySQL Server Severity:S3 (Non-critical)
Version:8.0 OS:Ubuntu (Ubuntu 18.04.3 LTS)
Assigned to: CPU Architecture:x86

[30 Oct 2019 10:10] Dmitry Smal
Description:
When running with insufficient memory MySQL 8.0 crashes with signal 11

====  VERSION  =======
root@rc1b-servername ~ # mysqld --version
/usr/sbin/mysqld  Ver 8.0.15-6 for debian-linux-gnu on x86_64 (Percona Server (GPL), Release '6', Revision '63abd08')

====  LOG =======
2019-10-29T03:11:01.181910+03:00 275188 [Warning] [MY-010055] [Server] IP address '49.51.12.244' could not be resolved: Name or service not known
2019-10-29T07:23:48.696248+03:00 4 [ERROR] [MY-010934] [Server] Out of memory (Needed 465429424 bytes)
2019-10-29T07:23:48.720946+03:00 4 [ERROR] [MY-010934] [Server] Out of memory; check if mysqld or some other process uses all available memory; if not, you may have to use 'ulimit' to allow mysqld to use more memory or you can add more swap space
2019-10-29T07:23:49.740330+03:00 4 [ERROR] [MY-010934] [Server] Out of memory (Needed 465429424 bytes)
2019-10-29T07:23:49.741105+03:00 4 [ERROR] [MY-010934] [Server] Out of memory; check if mysqld or some other process uses all available memory; if not, you may have to use 'ulimit' to allow mysqld to use more memory or you can add more swap space
04:23:49 UTC - mysqld got signal 11 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
Attempting to collect some information that could help diagnose the problem.
As this is a crash and something is definitely wrong, the information
collection process might fail.
Please help us make Percona Server better by reporting any
bugs at https://bugs.percona.com/

key_buffer_size=8388608
read_buffer_size=131072
max_used_connections=13
max_threads=200
thread_count=7
connection_count=6
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 87340 K  bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0x5626a82886a0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 7fe058252d90 thread_stack 0x30000
/usr/sbin/mysqld(my_print_stacktrace(unsigned char*, unsigned long)+0x3d) [0x5626a46f751d]
/usr/sbin/mysqld(handle_fatal_signal+0x423) [0x5626a39ea653]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x12890) [0x7fe06f3cf890]
/usr/sbin/mysqld(lex_start(THD*)+0x11) [0x5626a389aba1]
/usr/sbin/mysqld(dd::Update_dictionary_tables_ctx::Update_dictionary_tables_ctx(THD*)+0x2dd) [0x5626a44c07dd]
/usr/sbin/mysqld(bool dd::cache::Storage_adapter::store<dd::Event>(THD*, dd::Event*)+0xb2) [0x5626a453bda2]
/usr/sbin/mysqld(bool dd::cache::Dictionary_client::update<dd::Event>(dd::Event*)+0xd9) [0x5626a44eabd9]
/usr/sbin/mysqld(Event_db_repository::update_timing_fields_for_event(THD*, MYSQL_LEX_STRING, MYSQL_LEX_STRING, long, unsigned long long)+0x1ae) [0x5626a3b622de]
/usr/sbin/mysqld(Event_queue::get_top_for_execution_if_time(THD*, Event_queue_element_for_exec**)+0x1cf) [0x5626a3b6544f]
/usr/sbin/mysqld(Event_scheduler::run(THD*)+0x156) [0x5626a3b67246]
/usr/sbin/mysqld(+0x11ec534) [0x5626a3b67534]
/usr/sbin/mysqld(+0x1e27bbf) [0x5626a47a2bbf]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x76db) [0x7fe06f3c46db]
/lib/x86_64-linux-gnu/libc.so.6(clone+0x3f) [0x7fe06d3ec88f]

Trying to get some variables.
Some pointers may be invalid and cause the dump to abort.
Query (0): Connection ID (thread ID): 4
Status: NOT_KILLED

You may download the Percona Server operations manual by visiting
http://www.percona.com/software/percona-server/. You may find information
in the manual which will help you identify the cause of the crash.
2019-10-29T07:23:52.546698+03:00 0 [System] [MY-010116] [Server] /usr/sbin/mysqld (mysqld 8.0.15-6) starting as process 533187
2019-10-29T07:24:17.714586+03:00 1 [Warning] [MY-010005] [Server] Skip re-populating collations and character sets tables in read-only mode.
2019-10-29T07:24:18.030617+03:00 0 [System] [MY-010229] [Server] Starting crash recovery...
2019-10-29T07:24:18.045636+03:00 0 [System] [MY-010232] [Server] Crash recovery finished.
2019-10-29T07:24:18.465983+03:00 0 [System] [MY-010931] [Server] /usr/sbin/mysqld: ready for connections. Version: '8.0.15-6'  socket: '/var/run/mysqld/mysqld.sock'  port: 3306  Percona Server (GPL), Release '6', Revision '63abd08'.

=====MY.CNF ========
[client]
port = 3307
socket = /var/run/mysqld/mysqld.sock

[mysqld_safe]
nice = 0
socket = /var/run/mysqld/mysqld.sock
log_error = /var/log/mysql/error.log
timezone = Europe/Moscow
log_timestamps = SYSTEM
open_files_limit = 65535

[mysqld]
user = mysql
port = 3306
admin_port = 3307
admin_address = rc1b-servername
pid_file = /var/run/mysqld/mysqld.pid
plugin-dir = /usr/lib/mysql/plugin
socket = /var/run/mysqld/mysqld.sock
log_error = /var/log/mysql/error.log
general_log_file = /var/log/mysql/query.log
basedir = /usr
datadir = /var/lib/mysql
tmpdir = /var/lib/mysql/.tmp
lc_messages_dir	= /usr/share/mysql
max-connect-errors = 4294967295

ssl=ON
ssl-ca=/etc/mysql/ssl/allCAs.pem
ssl-cert=/etc/mysql/ssl/server.crt
ssl-key=/etc/mysql/ssl/server.key
require-secure-transport=ON

event_scheduler = ON
skip_external_locking = ON
default_storage_engine = InnoDB
disabled_storage_engines="MyISAM,MRG_MyISAM,BLACKHOLE,FEDERATED,ARCHIVE,CSV"
explicit_defaults_for_timestamp = ON
log_timestamps = SYSTEM
max_allowed_packet = 16777216
thread_stack = 192K

max_connections =  200
thread_handling = one-thread-per-connection
thread_cache_size = 5

# Slow log
slow_query_log_file = /var/log/mysql/slow-query.log

# InnoDB settings
innodb_buffer_pool_size = 2048M
innodb_file_per_table = ON
innodb_log_buffer_size = 16M
innodb_log_file_size = 256M
innodb_autoinc_lock_mode = 2
innodb_use_native_aio = ON
innodb_flush_method = O_DIRECT
innodb_flush_log_at_trx_commit = 1

# SQL mode
sql_mode = ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_ENGINE_SUBSTITUTION,NO_DIR_IN_CREATE

# Replication
server_id = 1
binlog_format = ROW
gtid_mode = ON
enforce_gtid_consistency = ON
log_bin_trust_function_creators = ON
log_slave_updates = ON
max_binlog_size = 100M
binlog_expire_logs_seconds = 86400
master_info_repository = TABLE
relay_log_info_repository = TABLE
relay_log_recovery = ON
log_bin = mysql-bin-log-rc1b-servername
log_bin_index = /var/lib/mysql/mysql-bin-log.index
relay_log = mysql-relay-log-rc1b-servername
relay_log_index = /var/lib/mysql/mysql-relay-log.index

# Semi-sync replication
plugin_load_add = 'rpl_semi_sync_master=semisync_master.so;rpl_semi_sync_slave=semisync_slave.so'
rpl_semi_sync_master_timeout = 31536000000
rpl_semi_sync_master_wait_for_slave_count = 1
rpl_semi_sync_master_wait_no_slave = ON
rpl_semi_sync_master_wait_point = AFTER_SYNC

read_only = ON
super_read_only = ON
slave_parallel_workers = 0
report_host = rc1b-servername
report_port = 3307

[mysqldump]
quick
quote-names
max_allowed_packet = 16M

##### TYPICAL MEMORY STATS ######
root@rc1b-servername ~ # free
              total        used        free      shared  buff/cache   available
Mem:        4046560     3195332      282260        4256      568968      608808
Swap:             0           0

##### QUERY #########
It's highly possible that crash occured during handling scheduled event:

CREATE DEFINER=`admin`@`localhost` EVENT `mdb_repl_mon_event` ON SCHEDULE EVERY 1 SECOND STARTS '2019-10-23 08:57:13' ON COMPLETION NOT PRESERVE ENABLE DO BEGIN
    SET SESSION innodb_lock_wait_timeout = 1;
    UPDATE mysql.mdb_repl_mon SET ts = CURRENT_TIMESTAMP(3) WHERE @@read_only = 0;
END

How to repeat:
No guaranteed way, try to run mysql with insufficient memory
[30 Oct 2019 10:43] MySQL Verification Team
Thank you for the bug report. We don't support Percona server, please try an MySQL Oracle server and report the results again Thanks.
[1 Dec 2019 1:00] Bugs System
No feedback was provided for this bug for over a month, so it is
being suspended automatically. If you are able to provide the
information that was originally requested, please do so and change
the status of the bug back to "Open".