Bug #103338 Starting mysqld on same data files sometimes errors out due to doublewrite buf
Submitted: 15 Apr 2021 23:37 Modified: 16 Apr 2021 6:36
Reporter: Manuel Ung Email Updates:
Status: Verified Impact on me:
None 
Category:MySQL Server: InnoDB storage engine Severity:S3 (Non-critical)
Version:8.0.23 OS:Any
Assigned to: CPU Architecture:Any

[15 Apr 2021 23:37] Manuel Ung 
Description:
The main.log_buffered-big test is flaky because when the test tries open a second instance of mysqld using the same data files while an existing instance is still running.

The expected behaviour is for the 2nd instance to block on the file lock on ibdata1. However, in 8.0.20 the doublewrite files were split into separate files, and we open and close the ibdata1 and doublewrite files in the same order.

This means that the first instance could have closed the ibdata1 file without closing the doublewrite files, while the second instance opens the ibdata1 file and attempts to also open the doublewrite files. Since there is no retry when opening doublewrite files, this just fails initialization.

How to repeat:
This was found because main.log_buffered-big was flaky.

Looking at the ./var/tmp/buffered1.err file, one can see

2021-04-15T23:20:19.884245Z 1 [ERROR] [MY-012574] [InnoDB] Unable to lock ./#ib_16384_0.dblwr error: 11
2021-04-15T23:20:19.884631Z 1 [Note] [MY-012575] [InnoDB] Check that you do not already have another mysqld process using the same InnoDB data or log files.
2021-04-15T23:20:19.884978Z 1 [ERROR] [MY-013538] [InnoDB] Doublewrite file create failed: ./#ib_16384_0.dblwr
2021-04-15T23:20:19.885347Z 1 [ERROR] [MY-012930] [InnoDB] Plugin initialization aborted at srv0start.cc[2195] with error I/O error.

Suggested fix:
Add retry when opening doublewrite files, or reorder how locking/unlocking of files are done.
[16 Apr 2021 6:36] MySQL Verification Team 
Hello Manuel Ung,

Thank you for the report and feedback.
Verified as described.

regards,
Umesh
[16 Oct 2024 12:57] Laurynas Biveinis 
main.log_buffered-big still flaky on 8.0.40
[14 May 7:51] Laurynas Biveinis 
Still flaky on 8.0.42
[27 Oct 16:22] Laurynas Biveinis 
bug 119247 is on the same testcase but the error message is different, looks like a different issue