Bug #111809 DEADLOCK of threads detected!
Submitted: 19 Jul 2023 8:06 Modified: 19 Jul 2023 12:41
Reporter: Yehuda Finkelshtein Email Updates:
Status: Not a Bug Impact on me:
None 
Category:MySQL Server Severity:S3 (Non-critical)
Version:8.33 OS:Linux
Assigned to: CPU Architecture:x86
Tags: DEADLOCK threads detected

[19 Jul 2023 8:06] Yehuda Finkelshtein
Description:
########################################
DEADLOCK of threads detected!
--Thread 140408121820928 has waited at trx0sys.h line 598 for 0 seconds the semaphore:
Mutex at 0x7fb69c34a7b8, Mutex TRX_SYS created trx0sys.cc:565, locked by 140408121820928

--Thread 140407597930240 has waited at trx0sys.h line 598 for 58 seconds the semaphore:
Mutex at 0x7fb69c34a7b8, Mutex TRX_SYS created trx0sys.cc:565, locked by 140408121820928

2023-07-19T07:26:01.242898Z 0 [ERROR] [MY-012982] [InnoDB] [FATAL] ######################################## Deadlock Detected!
2023-07-19T07:26:01.242916Z 0 [ERROR] [MY-013183] [InnoDB] Assertion failure: sync0arr.cc:678:ib::fatal triggered thread 140407916721920
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: http://dev.mysql.com/doc/refman/8.0/en/forcing-innodb-recovery.html
InnoDB: about forcing recovery.
2023-07-19T07:26:01Z UTC - mysqld got signal 6 ;
Most likely, you have hit a bug, but this error can also be caused by malfunctioning hardware.
BuildID[sha1]=2322a98ee785d298cbc399f95a31b9671531a40b
Thread pointer: 0x0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0 thread_stack 0x100000
/usr/sbin/mysqld(my_print_stacktrace(unsigned char const*, unsigned long)+0x41) [0x2124331]
/usr/sbin/mysqld(print_fatal_signal(int)+0x397) [0xfe7cb7]
/usr/sbin/mysqld(my_server_abort()+0x75) [0xfe7e05]
/usr/sbin/mysqld(my_abort()+0xe) [0x211e1be]
/usr/sbin/mysqld(ut_dbg_assertion_failed(char const*, char const*, unsigned long)+0x309) [0x236ae89]
/usr/sbin/mysqld(ib::fatal::~fatal()+0xcf) [0x236d7af]
/usr/sbin/mysqld() [0x23235e6]
/usr/sbin/mysqld(sync_array_detect_deadlock()+0x8a) [0x232395a]
/usr/sbin/mysqld(srv_error_monitor_thread()+0x1b9) [0x230b3f9]
/usr/sbin/mysqld(void Detached_thread::operator()<void (*)()>(void (*&&)())+0xca) [0x223a18a]
/lib64/libstdc++.so.6(+0xc2b13) [0x7fb6aee51b13]
/lib64/libpthread.so.0(+0x81ca) [0x7fb6b00c51ca]
/lib64/libc.so.6(clone+0x43) [0x7fb6ae469e73]
The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you find out what is causing the crash.
2023-07-19T07:26:04.523508Z 0 [Warning] [MY-010140] [Server] Could not increase number of max_open_files to more than 10000 (request: 10010)
2023-07-19T07:26:04.523513Z 0 [Warning] [MY-010142] [Server] Changed limits: table_open_cache: 3995 (requested 4000)
2023-07-19T07:26:04.705889Z 0 [Note] [MY-013932] [Server] BuildID[sha1]=2322a98ee785d298cbc399f95a31b9671531a40b
2023-07-19T07:26:04.705893Z 0 [Note] [MY-010949] [Server] Basedir set to /usr/.
2023-07-19T07:26:04.705901Z 0 [System] [MY-010116] [Server] /usr/sbin/mysqld (mysqld 8.0.33) starting as process 2596282

How to repeat:
running a long query that insert into temporary table
the query return 0 rows at the end but cause the database crash.

its happen on master and also on slave with super read only is on.

Suggested fix:
please advice how to overcome this issue since its cause downtime in our DB
[19 Jul 2023 12:41] MySQL Verification Team
Hi Mr. Finkelshtein,

Thank you for your bug report.

However, it is not a bug.

All transactional engines which are based on MVCC standard are checking for the deadlocks.

The fact that InnoDB has discovered this deadlock is a proof that InnoDB works perfectly.

There are various types of deadlocks, but the one into which you ran is of the type that necessitates the abortion of the running of the server. Otherwise, your data on the permanent storage would have been compromised.

Users should write their applications in a manner that the transaction which is rolled back is re-submitted.

This is all described in our Reference Manual.

Not a bug.
[16 Sep 2024 22:58] Marcos Albe
Dearest Sinisa,

This is not a row deadlock, which are totally expected; This is a mutex deadlock due to incorrect order of mutex/lock acquisition. How can you tell this is expected?????  A crashing server is expected?  Great publicity for your software.

This is a bug, 100% and still happens on 8.0.36:

2024-09-13T18:13:49.315611Z 0 [ERROR] [MY-012982] [InnoDB] [FATAL] ######################################## Deadlock Detected!
2024-09-13T18:13:49.315651Z 0 [ERROR] [MY-013183] [InnoDB] Assertion failure: sync0arr.cc:680:ib::fatal triggered thread 139959045433088

Please correct your assessment and re-open the bug.
Thanks!
[17 Sep 2024 8:39] MySQL Verification Team
Hi Mr. Albe,

This is expected behaviour.

Simply, one connection was waiting on something to be processed or on the end user input or similar.

Since we do not allow that application blocks mutex for more then 60 seconds, MySQL server generates itself a controlled assertion.

Beside the application error, another cause of these problems is usage of commodity hardware that can produce random errors. Those errors are not lasting for long time, so they can not be detected by hardware checking software.

Not a bug.
[17 Sep 2024 8:42] MySQL Verification Team
Hi,

Also this could be a case of corruption in the tablespace, which is usually caused by bad hardware.

We are always leaving a. possibility that there is a bug in our code.

However, let us inform you that this is a forum ONLY  for the reports with fully repeatable test cases.

Each of those test cases must consist of the set of SQL statements that always leads to the problem reported.

We do not have such a test case in this report.

When we do not have a test case, then the bug report can NOT be processed.

Not a bug.
[11 May 15:36] Eddy Ng
Hi.

i am getting similar error on 8.0.33.
i have a few questions.

1) doesn't this bug/crash generate stack trace for MySQL support to investigate further or simulate the issue?

2) i am suprise when you mention is NOT a bug. A deadlock is suppose to kill 1 of the offending query so that the other query can proceed and IS NOT suppose to crash the DB. So i am perplex why you say is not a bug?
[29 May 2:52] tao Zhou
I recently encountered the same crash error message on MySQL 8.0.28 version. Is there any solution available
--------------------------------------------------------------------
1852: 2025-05-28T19:06:37.308994+08:00 0 [ERROR] [MY-012982] [InnoDB] [FATAL] ######################################## Deadlock Detected!
1853: 2025-05-28T19:06:37.322153+08:00 0 [ERROR] [MY-013183] [InnoDB] Assertion failure: sync0arr.cc:688:ib::fatal triggered thread 140432478566144
1854: InnoDB: We intentionally generate a memory trap.
1855: InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
1856: InnoDB: If you get repeated assertion failures or crashes, even
1857: InnoDB: immediately after the mysqld startup, there may be
1858: InnoDB: corruption in the InnoDB tablespace. Please refer to
1859: InnoDB: http://dev.mysql.com/doc/refman/8.0/en/forcing-innodb-recovery.html
1860: InnoDB: about forcing recovery.
1861: 11:06:37 UTC - mysqld got signal 6 ;
1862: Most likely, you have hit a bug, but this error can also be caused by malfunctioning hardware.
1863: Thread pointer: 0x0
1864: Attempting backtrace. You can use the following information to find out
1865: where mysqld died. If you see no messages after this, something went
1866: terribly wrong...
1867: stack_bottom = 0 thread_stack 0x100000
1868: /data/mysql/base/8.0.28/bin/mysqld(my_print_stacktrace(unsigned char const*, unsigned long)+0x2e) [0x1f9ffde]
1869: /data/mysql/base/8.0.28/bin/mysqld(print_fatal_signal(int)+0x2eb) [0x103628b]
1870: /data/mysql/base/8.0.28/bin/mysqld(my_server_abort()+0x5e) [0x103638e]
1871: /data/mysql/base/8.0.28/bin/mysqld(my_abort()+0xa) [0x1f9a3aa]
1872: /data/mysql/base/8.0.28/bin/mysqld(ut_dbg_assertion_failed(char const*, char const*, unsigned long)+0x30c) [0x225c4bc]
1873: /data/mysql/base/8.0.28/bin/mysqld(ib::fatal::~fatal()+0x98) [0x225ee18]
1874: /data/mysql/base/8.0.28/bin/mysqld() [0x220cebc]
1875: /data/mysql/base/8.0.28/bin/mysqld(sync_array_detect_deadlock()+0x104) [0x220d1c4]
1876: /data/mysql/base/8.0.28/bin/mysqld(srv_error_monitor_thread()+0x19d) [0x21f504d]
1877: /data/mysql/base/8.0.28/bin/mysqld(std::thread::_State_impl<std::thread::_Invoker<std::tuple<Detached_thread, void (*)()> > >::_M_run()+0xbc) [0x212061c]
1878: /data/mysql/base/8.0.28/bin/mysqld() [0x279e1ef]
1879: /lib64/libpthread.so.0(+0x81cf) [0x7fbc24b831cf]
1880: /lib64/libc.so.6(clone+0x43) [0x7fbc22d26dd3]
1881: The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
1882: information that should help you find out what is causing the crash.