Bug #96819 Latching order violation during B-tree operations
Submitted: 10 Sep 2019 10:37 Modified: 10 Sep 2019 11:38
Reporter: Thirunarayanan Balathandayuthapani Email Updates:
Status: Not a Bug Impact on me:
None 
Category:MySQL Server: InnoDB storage engine Severity:S2 (Serious)
Version:5.7 OS:Any
Assigned to: CPU Architecture:Any
Tags: deadlock, innodb, SX lock

[10 Sep 2019 10:37] Thirunarayanan Balathandayuthapani
Description:
The following patch leads to thread deadlock in mysql-5.7. It turns out that
it is another case of both BUG#26225783 and Bug #80919.

diff --git a/storage/innobase/buf/buf0buf.cc b/storage/innobase/buf/buf0buf.cc
index ddfebaf..23baa6c 100644
--- a/storage/innobase/buf/buf0buf.cc
+++ b/storage/innobase/buf/buf0buf.cc
@@ -4617,6 +4617,9 @@ got_block:
                fix_block->made_dirty_with_no_latch = dirty_with_no_latch;
        }
 
+       rw_lock_x_lock(&block->lock);
+       rw_lock_x_unlock(&block->lock);
+
        mtr_memo_type_t fix_type;
 
        switch (rw_latch) {

The following messages are taken from error log:

########################################
DEADLOCK of threads detected!
rw-lock 0x7f4e85c9c840 --Thread 139975224411904 has waited at buf0buf.cc line 4620 for 0.00 seconds the semaphore:
X-lock on RW-latch at 0x7f4e85c9c840 created in file buf0buf.cc line 1460
a writer (thread id 139974972262144) has reserved it in mode  exclusive
number of readers 0, waiters flag 1, lock_word: 0
Last time read locked in file btr0cur.cc line 5697
Last time write locked in file /home/thiru/mysql-server/mysql-server/5.7/storage/innobase/include/mtr0mtr.ic line 153
Locked: thread 139974972262144 file mtr0mtr.ic line 153  X-LOCK
Locked: thread 139974972262144 file mtr0mtr.ic line 153  X-LOCK
rw-lock 0x7f4e85c502c8 --Thread 139974972262144 has waited at buf0buf.cc line 4620 for 0.00 seconds the semaphore:
X-lock (wait_ex) on RW-latch at 0x7f4e85c502c8 created in file buf0buf.cc line 1460
a writer (thread id 139974972262144) has reserved it in mode  wait exclusive
number of readers 1, waiters flag 0, lock_word: ffffffffefffffff
Last time read locked in file btr0cur.cc line 5736
Last time write locked in file /home/thiru/mysql-server/mysql-server/5.7/storage/innobase/include/mtr0mtr.ic line 117
Locked: thread 139975224411904 file btr0cur.cc line 5736  S-LOCK
Locked: thread 139975224411904 file btr0cur.cc line 5736  S-LOCK
2019-09-10T10:07:24.066654Z 0 [ERROR] [FATAL] InnoDB: ######################################## Deadlock Detected!

How to repeat:
./mtr --big-test --mysqld=--innodb-page-size=4k innodb.innodb_bulk_create_index{,,} --parallel=auto --repeat=5

Suggested fix:
Follow the latching order rules outlined in High Level Architecture of https://dev.mysql.com/worklog/task/?id=6326
[10 Sep 2019 11:38] Thirunarayanan Balathandayuthapani
I understand that debug patch which was pasted taking un-necessary x-lock on the page even though rw_latch is RW_NO_LATCH. So it doesn't violate the latching order. Withdrawing the bug report because of false alarm.