Bug #118814 I/O error while writing to file: .\#ib_16384_0.dblwr
Submitted: 12 Aug 3:47 Modified: 11 Oct 3:22
Reporter: Marci Assassin Email Updates:
Status: Open Impact on me:
None 
Category:MySQL Server: Windows Severity:S2 (Serious)
Version:8.0.39 OS:Windows
Assigned to: CPU Architecture:Any

[12 Aug 3:47] Marci Assassin
Description:
In the production environment, I discovered that the mysqld service automatically shut down.

The following logs are extracted from the .err file (timezone: UTC+9), with four main segments:

[1] 14:12:16 - First occurrence of write failure to #ib_16384_0.dblwr file
[2] 2025-08-08 23:17:11 0x1868 INNODB MONITOR OUTPUT - First monitoring output
[3] 2025-08-08 23:28:42 0x1868 INNODB MONITOR OUTPUT - Final monitoring output
[4] Log entries immediately preceding mysqld service termination

This section describes only the content of the fourth segment. Additional log information will be submitted as an attachment.

[4]

2025-08-08T14:28:47.129217Z 0 [Warning] [MY-012638] [InnoDB] Retry attempts for writing partial data failed.
2025-08-08T14:28:47.129462Z 0 [ERROR] [MY-013676] [InnoDB] I/O error while writing to file: .\#ib_16384_0.dblwr. Retrying ...
InnoDB: ###### Diagnostic info printed to the standard error stream
2025-08-08T14:28:50.891291Z 0 [ERROR] [MY-012872] [InnoDB] [FATAL] Semaphore wait has lasted > 600 seconds. We intentionally crash the server because it appears to be hung.
2025-08-08T14:28:50.897769Z 0 [ERROR] [MY-013183] [InnoDB] Assertion failure: srv0srv.cc:1879:ib::fatal triggered thread 6240
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: http://dev.mysql.com/doc/refman/8.0/en/forcing-innodb-recovery.html
InnoDB: about forcing recovery.
2025-08-08T14:28:50Z UTC - mysqld got exception 0x16 ;
Most likely, you have hit a bug, but this error can also be caused by malfunctioning hardware.
Thread pointer: 0x0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
2025-08-08T14:28:57.144929Z 0 [Warning] [MY-012638] [InnoDB] Retry attempts for writing partial data failed.
2025-08-08T14:28:57.145175Z 0 [ERROR] [MY-013676] [InnoDB] I/O error while writing to file: .\#ib_16384_0.dblwr. Retrying ...
7ff7e49ce318    mysqld.exe!my_print_stacktrace()[stacktrace.cc:430]
7ff7e3b41751    mysqld.exe!print_fatal_signal()[signal_handler.cc:159]
7ff7e3b41493    mysqld.exe!my_server_abort()[signal_handler.cc:270]
7ff7e49b293a    mysqld.exe!my_abort()[my_init.cc:264]
7ff7e4b83729    mysqld.exe!ut_dbg_assertion_failed()[ut0dbg.cc:100]
7ff7e4a96973    mysqld.exe!ib::fatal::~fatal()[ut0ut.cc:523]
7ff7e4a61e56    mysqld.exe!srv_error_monitor_thread()[srv0srv.cc:1882]
7ff7e4a47a3e    mysqld.exe!Detached_thread::operator()<void (__cdecl*)(void)>()[os0thread-create.h:193]
7ff7e4a47bc5    mysqld.exe!std::thread::_Invoke<std::tuple<Detached_thread,void (__cdecl*)(void)>,0,1>()[thread:56]
7ffefacb268a    ucrtbase.dll!_o_exp()
7ffefd9d7974    KERNEL32.DLL!BaseThreadInitThunk()
7ffefe6da371    ntdll.dll!RtlUserThreadStart()
The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you find out what is causing the crash.

How to repeat:
We encountered this issue unexpectedly while using MySQL via Entity Framework Core (EF Core), without any explicit row or table locking in place. The server is a Dell PowerEdge R250 configured with RAID 1 storage.
[12 Aug 17:43] MySQL Verification Team
Did you check the Event Viewer system logs to see for any Ntfs or disk device errors?

Disable antivirus scanners on the datadir too.

-- 
Shane Bester, MySQL Senior Principal Technical Support Engineer
Oracle Corporation
http://dev.mysql.com/
[10 Sep 1:39] Marci Assassin
Approximately two weeks after the initial occurrence, we observed the same phenomenon again.  
During the first occurrence, no disk failures were recorded in the Windows event logs. However, during the second occurrence, we observed messages related to disk write failures.  

We have now replaced the hard drive and are monitoring whether the issue reoccurs.
[11 Sep 13:08] MySQL Verification Team
Let us know if status changes but this looks like hardware issue.
[24 Sep 4:02] Marci Assassin
After replacing the hard drive, we encountered the same issue again. In the production environment, the customer insists that antivirus software must be installed. Therefore, we were unable to test whether the issue would be resolved if the antivirus software was disabled.

We were able to successfully recover the database using the following method. Can the possible cause be inferred based on this recovery approach?

Recovery steps:

1.Add the following configurations to my.ini:
innodb_force_recovery=4
skip_innodb_doublewrite

2. Run the following command:
mysqld --initialize-insecure --user=mysql

3. Remove the configurations added in Step 1.
[25 Sep 17:08] MySQL Verification Team
Hi,

2025-09-12T12:09:20.385603Z 0 [Warning] [MY-012638] [InnoDB] Retry attempts for writing partial data failed.
2025-09-12T12:09:20.392129Z 0 [ERROR] [MY-012639] [InnoDB] Write to file .\#ib_16384_0.dblwr failed at offset 0, 65536 bytes should have been written, only 0 were written. Operating system error number 0. Check that your OS and file system support files of this size. Check also that the disk is not full or a disk quota exceeded.
2025-09-12T12:09:20.392786Z 0 [ERROR] [MY-012640] [InnoDB] Error number 0 means 'No error'
2025-09-12T12:09:20.392973Z 0 [ERROR] [MY-013676] [InnoDB] I/O error while writing to file: .\#ib_16384_0.dblwr. Retrying ...
...

This is what your OS returned to MySQL. We cannot work around your OS. 
Why is your OS making this problem is not something we can guess.

This is especially problematic as "Operating system error number 0" is "no error" so we have windows returning a no error while failing to allow file to be written. 

In case of antivirus issue normally we'd see ERROR_ACCESS_DENIED (5) or ERROR_SHARING_VIOLATION (32) not error 0.

What is possible is that we have a bug in our code that's incorrectly reading the error Windows returned, so that actual error is not 0 but something else (5 for e.g.) but it is def. something outside of our hands that is preventing this write to happen.
[26 Sep 9:56] MySQL Verification Team
Hi,

I discussed with my colleagues and the error message here is not useful as the error happens earlier in code. I'll verify the bug as this code needs to be re-checked, but with regards to your problem it is most probably a windows issue and not innodb issue.
Code inspection shows that WriteFile or GetOverlappedResult has failed, but we don't really know how it failed. It can very well be due to antivirus but we cannot be sure.

In any way please share some info about your setup, please upload full config file, we especially want to see 
innodb-page-size
innodb-doublewrite
innodb_doublewrite_pages
innodb_write_io_threads