Description:
Hello,
server had uptime (and mysqld too) more than 1month.
2.10.2024 at approx. 12:57 (GMT+2) mysqld stoped. It caused 100% system load (but CPU jumps and cpu usage were normal). Also RAM was around 60% of usage.
I wasn't able to acces that server (terminal and ssh). Finaly we had to reset whole VM.
I sent logs from iDRAC to Dell and they told me, that there is no HW issue.
On that server, there are another 14 VMs which are running smoothly.
I went through the journalctl log and there is no error in that date and time.
In MySQL error log, there are lot of semaphores wait like this (only for that 1 day - day before and also day after incident, there are no errors):
2024-10-02T10:58:13.558389Z 0 [Warning] [MY-012985] [InnoDB] A long semaphore wait:
--Thread 140596882601536 has waited at log0meb.cc line 1781 for 242 seconds the semaphore:
Mutex at 0x7fe192101e80, Mutex LOG_FILES created log0log.cc:642, locked by 140597131777600
At the end of the log, there are theese informations:
InnoDB: ###### Starts InnoDB Monitor for 30 secs to print diagnostic info:
InnoDB: Pending preads 0, pwrites 0
InnoDB: ###### Diagnostic info printed to the standard error stream
2024-10-02T11:09:34.813509Z 0 [ERROR] [MY-012872] [InnoDB] [FATAL] Semaphore wait has lasted > 600 seconds. We intentionally crash the server because it appears to be hung.
2024-10-02T11:09:34.813898Z 0 [ERROR] [MY-013183] [InnoDB] Assertion failure: srv0srv.cc:1879:ib::fatal triggered thread 140597081421376
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: http://dev.mysql.com/doc/refman/8.0/en/forcing-innodb-recovery.html
InnoDB: about forcing recovery.
2024-10-02T11:09:34Z UTC - mysqld got signal 6 ;
Most likely, you have hit a bug, but this error can also be caused by malfunctioning hardware.
BuildID[sha1]=d40fc297f92a0ddfd99ce83c39d23520b7a3cb37
Thread pointer: 0x0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0 thread_stack 0x100000
/usr/sbin/mysqld(my_print_stacktrace(unsigned char const*, unsigned long)+0x41) [0x55ca6794f961]
/usr/sbin/mysqld(print_fatal_signal(int)+0x3bc) [0x55ca66f5d1bc]
/usr/sbin/mysqld(my_server_abort()+0x7e) [0x55ca66f5d2ee]
/usr/sbin/mysqld(my_abort()+0xe) [0x55ca67949cfe]
/usr/sbin/mysqld(ut_dbg_assertion_failed(char const*, char const*, unsigned long)+0x183) [0x55ca67b05523]
/usr/sbin/mysqld(+0x17be5c6) [0x55ca67b1d5c6]
/usr/sbin/mysqld(srv_error_monitor_thread()+0x5ae) [0x55ca67ac4b3e]
/usr/sbin/mysqld(+0x16cad25) [0x55ca67a29d25]
/lib/x86_64-linux-gnu/libstdc++.so.6(+0xdc253) [0x7fe1a770d253]
/lib/x86_64-linux-gnu/libc.so.6(+0x94ac3) [0x7fe1a7395ac3]
/lib/x86_64-linux-gnu/libc.so.6(+0x126850) [0x7fe1a7427850]
The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you find out what is causing the crash.
After reset, everything seems to be normal, no errors in that log, etc.
Can you help me with that?
Thank you!
How to repeat:
I don't know.
