Bug #100768 MySQL Server 8 is crashed and rebooted automatically with a long semaphore wait
Submitted: 8 Sep 2020 11:40 Modified: 8 Sep 2020 18:57
Reporter: Rufus Martin Email Updates:
Status: Can't repeat Impact on me:
None 
Category:MySQL Server: InnoDB storage engine Severity:S2 (Serious)
Version: OS:CentOS (version 8)
Assigned to: MySQL Verification Team CPU Architecture:Any
Tags: innodb, readlock

[8 Sep 2020 11:40] Rufus Martin
Description:
Team,

We are running MySQLDump process on every weekend. We have some TB of MySQL data. So the dump & zip process will run for more than a day. We faced this kind of error on last 2 runs of mysql dump process. At some point of mysqldump process we got error as below:

InnoDB: ###### Diagnostic info printed to the standard error stream
2020-08-29T22:46:23.549417Z 0 [Warning] [MY-012985] [InnoDB] A long semaphore wait:
--Thread 139651551487744 has waited at trx0rec.cc line 2302 for 271.00 seconds the semaphore:
S-lock on RW-latch at 0x7f2ba848c3d8 created in file trx0purge.cc line 218
a writer (thread id 139825603688192) has reserved it in mode  exclusive
number of readers 0, waiters flag 1, lock_word: 0
Last time read locked in file trx0rec.cc line 2302
Last time write locked in file ../../../mysql-8.0.15/storage/innobase/trx/trx0purge.cc line 2170
2020-08-29T22:46:23.549473Z 0 [Warning] [MY-012985] [InnoDB] A long semaphore wait:
--Thread 139651546769152 has waited at trx0trx.cc line 1262 for 272.00 seconds the semaphore:
Mutex at 0x7f2ba8399588, Mutex TRX_SYS created trx0sys.cc:387, lock var 1

2020-08-29T22:46:23.549518Z 0 [Warning] [MY-012985] [InnoDB] A long semaphore wait:
--Thread 139632290690816 has waited at read0read.cc line 546 for 265108.00 seconds the semaphore:
	Mutex at 0x7f2ba8399588, Mutex TRX_SYS created trx0sys.cc:387, lock var 1

.......

InnoDB: Progress in percents: 12020-08-29T23:34:11.534514Z 0 [System] [MY-010229] [Server] Starting crash recovery...
2020-08-29T23:34:11.544694Z 0 [System] [MY-010232] [Server] Crash recovery finished

After 45 mins the server will goes into crash recovery. 
We didn't find exact reason for this issue.
Can someone can help us here?

How to repeat:
This issue is repeating for every week (Whenever we are running mysql dump process)
[8 Sep 2020 18:57] MySQL Verification Team
Hi,

I cannot reproduce this. I suggest you double check your hardware, you might be overloading your storage subsystem or you might have hw error in your storage subsystem. What does your system log show for this period when you have a crash?

A suggestion for TB's of backup - use mysqlbackup and not mysqldump.

all best
Bogdan