MySQL Bugs: #111487: File .\undo_002: 'aio write' returned OS error 1560. Cannot continue operation

Bug #111487	File .\undo_002: 'aio write' returned OS error 1560. Cannot continue operation
Submitted:	19 Jun 2023 11:55	Modified:	29 Jun 2023 16:57
Reporter:	Marc Kessman	Email Updates:
Status:	Not a Bug	Impact on me:	None
Category:	MySQL Server: Windows	Severity:	S3 (Non-critical)
Version:	8.0.31	OS:	Windows (2019 Datacenter)
Assigned to:		CPU Architecture:	x86 (Intel)
Tags:	1560

Description:
1) The server did NOT crash running a specific query.

2) The log file generated an error "Operating system error number 1460 in a file operation." and "Assertion failure: os0file.cc:7022:slot->is_reserved thread 9968"

3) The MySQL attempted a backtrace and came back with " File .\undo_002: 'aio write' returned OS error 1560. Cannot continue operation"

4) The MySQL instance crashed.

5) I expected MySQl would recover from this operation but it did not.

6) We have searched your bug database and stack over flow, no one has this issue

How to repeat:
It happens once a week but there is no way to reproduce. CPU spikes to
100% and then MySQL crashes. We then have to stop and start the service
for it to recover. Brand new server 24GB ram with SSD drives 4 CPU
Sockets and 16 Virtual Processors running on Hypervison with average CPU
of 28%

Hi Mr. Kessman,

Thank you for your  bug report.

However, this is not a bug in MySQL. This error shows that something is wrong with your hardware or your Windows installation or configuration.

Also, you are very vague about whether this is Windows error 1460 or 1560.

Anyway, Microsoft has solution for both of them and neither is in the realm of our server.

These are the links:

https://social.technet.microsoft.com/Forums/ie/en-US/059a64b2-199b-426b-a271-a2163ff09cb8/...

and:

https://social.msdn.microsoft.com/Forums/en-US/39881de8-cd5d-436f-8732-40f40e2e468c/error-...

The last error code is very problematic in Windows, as it is used by different Windows modules, each with a different meaning.

Not a bug.

This is a bug as other users have reported the same issue to you. Your advise was to rename the jemalloc to force the system to use the old malloc.

Well that didn't work either, we just had another server crash with the following in the log file:

2023-06-28T09:32:41.005454Z 0 [ERROR] [MY-012611] [InnoDB] Operating system error number 1460 in a file operation.

2023-06-28T09:32:41.005909Z 0 [Note] [MY-012618] [InnoDB] Refer to your operating system documentation for operating system error code information.

2023-06-28T09:32:41.006334Z 0 [ERROR] [MY-012646] [InnoDB] File .\undo_001: 'aio write' returned OS error 1560. Cannot continue operation

2023-06-28T09:32:41.006718Z 0 [ERROR] [MY-012981] [InnoDB] Cannot continue operation.

2023-06-28T11:07:25.704032Z 0 [Note] [MY-013848] [Server] jemalloc.dll not found. Falling back to default malloc.

According to:
https://learn.microsoft.com/en-us/windows/win32/debug/system-error-codes--1300-1699-

The error 1460 means:

--
ERROR_TIMEOUT
    1460 (0x5B4)
    This operation returned because the timeout period expired.
--

Is this a local disk drive or a network share or something else clustered?
Are there any relevant entries in the event viewer logs on the system at the time?

I do see:
 https://learn.microsoft.com/en-us/troubleshoot/windows-server/backup-and-storage/event-512...

InnoDB adds 100 to the error numbers,   so we can ignore 1560 and just focus on 1460.

The drives are local SSD. 

Everything is installed locally.

No entries in either the Windows Application or System Logs.

Also, the MySQL service is set to restart on the recovery tab, but it never does. It just hard crashes and requires us to reboot the server as you can't even start the service after a crash.

It happens every 3 days now.

Hi,

The timeout that is in question is  a problem in the operating system, which is Windows.

It is the error that comes from OS and not from our code.

Hence, you should enable Windows diagnostics in order to get what is wrong with your OS or it's setup.

Also, many other programs on Windows return that error message from the OS, like MS Live Mail, MS Edge and many others.

Simply, this is problem with Windoes or its setup. Some articles even note that this can happen after the upgrade of the OS. In that case, Microsoft is obliged to provide another patch.

Also, Microsoft documentation claims that this is due to the corrupt driver or filesystem. 

In either case, this is not MySQL bug.

How can you say this is NOT a bug when your company posted and say it IS a bug. Look at the URL below:

https://support.oracle.com/knowledge/Oracle%20Database%20Products/2904830_1.html

Windows Servers Crashing in jemalloc.dll and libcrypto-1_1-x64.dll (Doc ID 2904830.1)
Last updated on NOVEMBER 25, 2022

And here is another bug report

https://bugs.mysql.com/bug.php?id=103133

Hi,

Yes, you are correct. The bug is in those two libraries that you are mentioning.

However, neither of those two libraries is made by us. Both are made by Microsoft. We can not fix something which is not our code and for which we do not have a source code.

That is why our bug, that you mentioned can not be fixed, since the bug is in the Windows OS. You can not read that, since this is internal information. Simply, that bug was waiting for he new versions of those Microsoft libraries, but Microsoft has not addressed those issues in the newer versions of the said libraries.

Not a bug.

Hi,

Yes, you are right, but it was you who brought up the problem with those MS libraries.

Your problem is with some patch in your OS. This MS problem is discussed on many sites.

Timeouts in the operating system are not under control of the running applications.

Not a bug.

So it's obvious that there is an issue with the MS Libraries pertaining/working with Oracle MySQL v8.0.30/34 because I uninstalled v8.0.30/34 and installed 5.7 and there are no issues. No crashes. I put a ticket in with MS and they point the finger back to you (Oracle) stating there is an issue in your binary that is not properly calling the memory allocation routines.

Great, now I am stuck between two billion dollar companies pointing the finger at each other. I thought those days were over but I guess not.

Hence, are you telling the community here that we

1) Either have to wait for Microsoft to fix their libraries
2) Uninstall v8.0 and install the old 5.7
3) Oracle gives it's customers a workaround or hotfix so we can continue using v8.0

Even WAMP SERVER who puts together Windows builds Apache, PHP and MySQl bundles says it's a bug.

So the entire community out here says it's an Oracle bug including Microsoft, but you still say it's not.

Maybe you should take another look at what the community is showing you and re-investigate the issue instead of keep saying it's not your "bug" and treat it like a hot potato.

Hi,

MySQL has in it's code many calls that might result in a timeout. Whenever there is a timeout, we return our own error message for the timeout.

But, this is none of our codes.

All Windows (or Linux) system libraries and kernels have there own operations which might result in the timeout. When those operations get timeout, where it. should not happen, they return their own code for timeout. Hence, this error is from OS, in the part of our code where timeout is not supposed to happen.

That is why LiveMail and Bing users also wait for the the patch from Microsoft, that will fix the problem.

Please, do remember, that this is a forum for the bugs with fully repeatable test cases.

Where is the test case that you have sent us, which would enable us to diagnose the problem ????