Description:
Hello,
a couple of days ago, one of our MySQL instances on AWS (RDS) failed.
We don't have much information about it; it seems to have been caused by too many files open.
However, I'm reporting it because the instance log had a failed assertion, and it explicitly mentioned to report the bug.
We're dealing in parallel with the AWS support to get more context, so if you need more information, let me know and I'll see what I can do.
Content of the log:
2020-02-17T17:38:06.745893Z 0 [ERROR] [MY-010283] [Server] Error in accept: Too many open files
2020-02-17T17:43:24.309244Z 26983118 [ERROR] [MY-013132] [Server] The table '/rdsdbdata/tmp/#sql46b6_19bbace_31a' is full!
2020-02-17T17:43:31.354540Z 26983519 [ERROR] [MY-013132] [Server] The table '/rdsdbdata/tmp/#sql46b6_19bbc5f_4a4' is full!
2020-02-17T17:43:56.960186Z 26983610 [ERROR] [MY-013132] [Server] The table '/rdsdbdata/tmp/#sql46b6_19bbcba_162' is full!
2020-02-17T17:44:00.530367Z 26968364 [ERROR] [MY-012592] [InnoDB] Operating system error number 24 in a file operation.
2020-02-17T17:44:00.530409Z 26968364 [ERROR] [MY-012596] [InnoDB] Error number 24 means 'Too many open files'
2020-02-17T17:44:00.530425Z 26968364 [ERROR] [MY-012646] [InnoDB] File ./<database_name>/<table_name>.ibd: 'open' returned OS error 124. Cannot continue operation
2020-02-17T17:44:00.531388Z 26968364 [ERROR] [MY-012981] [InnoDB] Cannot continue operation.
2020-02-17T17:44:00.571692Z 26987494 [ERROR] [MY-013183] [InnoDB] Assertion failure: log0buf.cc:700:srv_shutdown_state <= SRV_SHUTDOWN_FLUSH_PHASE thread 47209235810048
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: http://dev.mysql.com/doc/refman/8.0/en/forcing-innodb-recovery.html
InnoDB: about forcing recovery.
17:44:00 UTC - mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
Attempting to collect some information that could help diagnose the problem.
As this is a crash and something is definitely wrong, the information
collection process might fail.
key_buffer_size=1048576
read_buffer_size=262144
max_used_connections=157
max_threads=600
thread_count=132
connection_count=131
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 314889 K bytes of memory
Hope that's ok; if not, decrease some variables in the equation.
Thread pointer: 0x2afb4cffd000
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 2aefc1bafd00 thread_stack 0x40000
/rdsdbbin/mysql/bin/mysqld(my_print_stacktrace(unsigned char*, unsigned long)+0x2e) [0x1d3154e]
/rdsdbbin/mysql/bin/mysqld(handle_fatal_signal+0x413) [0xf34793]
/lib64/libpthread.so.0(+0xf100) [0x2aee6761c100]
/lib64/libc.so.6(gsignal+0x37) [0x2aee685a15f7]
/lib64/libc.so.6(abort+0x148) [0x2aee685a2ce8]
/rdsdbbin/mysql/bin/mysqld(ut_dbg_assertion_failed(char const*, char const*, unsigned long)+0x2c6) [0x20dd846]
/rdsdbbin/mysql/bin/mysqld(log_buffer_reserve(log_t&, unsigned long)+0x250) [0x1f87f90]
/rdsdbbin/mysql/bin/mysqld(mtr_t::Command::execute()+0x47) [0x1fbe7c7]
/rdsdbbin/mysql/bin/mysqld(mtr_t::commit()+0x69) [0x1fbe9d9]
/rdsdbbin/mysql/bin/mysqld(trx_undo_report_row_operation(unsigned long, unsigned long, que_thr_t*, dict_index_t*, dtuple_t const*, upd_t const*, unsigned long, unsigned char const*, unsigned long const*, unsigned long*)+0x582) [0x20b9502]
/rdsdbbin/mysql/bin/mysqld(btr_cur_update_in_place(unsigned long, btr_cur_t*, unsigned long*, upd_t const*, unsigned long, que_thr_t*, unsigned long, mtr_t*)+0x16e) [0x21145de]
/rdsdbbin/mysql/bin/mysqld(btr_cur_optimistic_update(unsigned long, btr_cur_t*, unsigned long**, mem_block_info_t**, upd_t const*, unsigned long, que_thr_t*, unsigned long, mtr_t*)+0x261) [0x2114e21]
2020-02-17T17:44:00.631596Z 0 [ERROR] [MY-013183] [InnoDB] Assertion failure: log0buf.cc:700:srv_shutdown_state <= SRV_SHUTDOWN_FLUSH_PHASE thread 47203674490624
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: http://dev.mysql.com/doc/refman/8.0/en/forcing-innodb-recovery.html
InnoDB: about forcing recovery.
InnoDB: Progress in percents: 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 1002020-02-17T17:44:17.290757Z 0 [Warning] [MY-010101] [Server] Insecure configuration for --secure-file-priv: Location is accessible to all OS users. Consider choosing a different directory.
2020-02-17T17:44:17.290800Z 0 [System] [MY-010116] [Server] /rdsdbbin/mysql/bin/mysqld (mysqld 8.0.15) starting as process 17434
2020-02-17T17:44:52.976753Z 0 [System] [MY-010229] [Server] Starting crash recovery...
2020-02-17T17:44:52.985864Z 0 [System] [MY-010232] [Server] Crash recovery finished.
2020-02-17T17:44:53.215976Z 0 [System] [MY-010931] [Server] /rdsdbbin/mysql/bin/mysqld: ready for connections. Version: '8.0.15' socket: '/tmp/mysql.sock' port: 3306 Source distribution.
2020-02-17T17:44:53.386553Z 0 [System] [MY-011323] [Server] X Plugin ready for connections. Socket: '/tmp/mysqlx.sock' bind-address: '::' port: 33060
How to repeat:
This was a once-off situation; likely can't reproduce.
