Bug #108228 MySQL Router 8.0.30 crashes on Windows
Submitted: 23 Aug 2022 1:37 Modified: 18 Apr 2023 12:22
Reporter: Keith Lammers Email Updates:
Status: Closed Impact on me:
None 
Category:MySQL Router Severity:S2 (Serious)
Version:8.0.30 OS:Windows (Server 2022 Core)
Assigned to: CPU Architecture:x86

[23 Aug 2022 1:37] Keith Lammers
Description:
No issues with MySQL Router 8.0.28 on these machines. Upgraded to 8.0.30 and since then the MySQL Router service has been randomly crashing on both machines.

There are a couple of warnings in the MySQL Router log, but no errors. The Windows System Event log shows this entry at the time of the crash:

Faulting application name: mysqlrouter.exe, version: 8.0.30.0, time stamp: 0x62c60deb
Faulting module name: mysqlrouter_routing.dll, version: 0.0.0.0, time stamp: 0x62c60e02
Exception code: 0xc00000fd
Fault offset: 0x0000000000025aa2
Faulting process id: 0xa3c
Faulting application start time: 0x01d8b65cf61a5203
Faulting application path: C:\Program Files\MySQL\MySQL Router 8.0\bin\mysqlrouter.exe
Faulting module path: C:\Program Files\MySQL\MySQL Router 8.0\bin\mysqlrouter_routing.dll
Report Id: 99d28ba8-4e29-4411-b70c-5ff949a80a33
Faulting package full name: 
Faulting package-relative application ID: 

How to repeat:
The crashes are seemingly random, but these are the changes I made to the environment before the crashes started happening:

* Upgraded MySQL Router from 8.0.28 on 8.0.30 (Windows Server 2022 Core)
* Upgraded all MySQL Server instances (Ubuntu Server 20.04) from 8.0.28 to 8.0.30
* MySQL Router crashed randomly multiple times on both Windows servers.
[24 Aug 2022 7:24] MySQL Verification Team
Hi,

Having issue reproducing this. 

Do you run some antivirus on those machines? You did not notice any regularity when the crash happen, time, load, anything? Do you, maybe, monitor the load on those windows servers?

Thanks
[24 Aug 2022 13:27] Keith Lammers
Replied with the details in a private comment.
[25 Aug 2022 3:28] MySQL Verification Team
Hi,

Do not use MySQL Server 8.0.29 please. You can test 8.0.29 router.

If I understood properly you moved the router to .28 and left Server at .30 and the crashes stopped. I cannot still reproduce crashes locally, please let me know if you move to .29 or .30 (router only not changing server) and the crashes start again.

Please send me the full config (router and server and connect string you are using from your app) so I can try to reproduce this again using configuration as similar as possible to yours

Thanks
[25 Aug 2022 15:44] Keith Lammers
Replied in a private comment, thanks!
[26 Aug 2022 6:26] MySQL Verification Team
Thanks for all the details Keith,

I hope we will manage to figure out what is making this crashes happen between .28 and .29. 

have a nice vacation
[8 Sep 2022 18:57] Keith Lammers
I'm available for more testing if needed. Would it help if I tried to reproduce it using sysbench to hopefully make it more repeatable?

Also, if there's a way for me to have a crash dump generated for mysqlrouter.exe when it dies, let me know and I'll try to capture that as well.

Thanks!
[9 Sep 2022 8:29] Jan Kneschke
Posted by developer:
 
It would help to start the 8.0.30 router with the --core-file.

$ mysqlrouter -c {configfile} --core-file

continue with the testing until it crashes, and check for a *.dmp file (a minidump).
[13 Sep 2022 13:40] Keith Lammers
I tried this out but it appears mysqlrouter.exe on Windows doesn't have a --core-file option available. Is there an equivalent on Windows?
[12 Oct 2022 12:22] Pawel Mroszczyk
Posted by developer:
 
Hi Keith,

I believe the --core-file functionality has just become available on Windows in Router 8.0.31.  Could you give it a try?  I see Router 8.0.31 is already available on dev.mysql.com.

For your convenience, here's additional info:
https://dev.mysql.com/doc/mysql-router/8.0/en/mysql-router-conf-options.html#option_mysqlr...
[12 Oct 2022 14:19] Keith Lammers
MySQL Router 8.0.31 seems to have started successfully with the --core-file option. I will update this ticket when it crashes next with the core file.
[13 Oct 2022 14:13] Keith Lammers
Can confirm that MySQL Router 8.0.31 still crashes on Windows, but it did not generate a crash dump even though it was running with the --core-file option. I searched the entire drive for *.dmp files, and there aren't any for MySQL Router.
[13 Oct 2022 16:57] MySQL Verification Team
Windows can be configured to generate crash dumps,  maybe a full crash dump will help if repeatability is an issue.

https://learn.microsoft.com/en-us/windows/win32/wer/collecting-user-mode-dumps

mysqlslap with a high --iterations and --concurrency with a low --number-of-queries  will stress test the connections code..
[29 Mar 2023 17:52] Keith Lammers
I completely missed your last reply on this, apologies for the delay! I will see if I can get a user-mode dump for the crash.
[3 Apr 2023 19:00] Keith Lammers
I've been running 8.0.32 on the affected machine since Thursday and haven't had any crashes yet. I will monitor for another week and report back.
[10 Apr 2023 19:05] Keith Lammers
MySQL Router 8.0.32 has been running on the affected machines for over a week now with no crashing issues, so I think it's safe to mark this as resolved :)
[18 Apr 2023 12:05] Jan Kneschke
Posted by developer:
 
Confirmed to be fixed in at least 8.0.32 by reporter.
[21 Apr 2023 9:55] Edward Gilmore
Posted by developer:
 
Added the following note to the MySQL Router 8.0.33 release notes:
	
  Under certain circumstances, on Microsoft Windows platforms, MySQL Router crashed repeatedly.