MySQL Bugs: #108228: MySQL Router 8.0.30 crashes on Windows

Bug #108228	MySQL Router 8.0.30 crashes on Windows
Submitted:	23 Aug 2022 1:37	Modified:	18 Apr 2023 12:22
Reporter:	Keith Lammers	Email Updates:
Status:	Closed	Impact on me:	None
Category:	MySQL Router	Severity:	S2 (Serious)
Version:	8.0.30	OS:	Windows (Server 2022 Core)
Assigned to:		CPU Architecture:	x86

Description:
No issues with MySQL Router 8.0.28 on these machines. Upgraded to 8.0.30 and since then the MySQL Router service has been randomly crashing on both machines.

There are a couple of warnings in the MySQL Router log, but no errors. The Windows System Event log shows this entry at the time of the crash:

Faulting application name: mysqlrouter.exe, version: 8.0.30.0, time stamp: 0x62c60deb
Faulting module name: mysqlrouter_routing.dll, version: 0.0.0.0, time stamp: 0x62c60e02
Exception code: 0xc00000fd
Fault offset: 0x0000000000025aa2
Faulting process id: 0xa3c
Faulting application start time: 0x01d8b65cf61a5203
Faulting application path: C:\Program Files\MySQL\MySQL Router 8.0\bin\mysqlrouter.exe
Faulting module path: C:\Program Files\MySQL\MySQL Router 8.0\bin\mysqlrouter_routing.dll
Report Id: 99d28ba8-4e29-4411-b70c-5ff949a80a33
Faulting package full name: 
Faulting package-relative application ID: 

How to repeat:
The crashes are seemingly random, but these are the changes I made to the environment before the crashes started happening:

* Upgraded MySQL Router from 8.0.28 on 8.0.30 (Windows Server 2022 Core)
* Upgraded all MySQL Server instances (Ubuntu Server 20.04) from 8.0.28 to 8.0.30
* MySQL Router crashed randomly multiple times on both Windows servers.

Hi,

Having issue reproducing this. 

Do you run some antivirus on those machines? You did not notice any regularity when the crash happen, time, load, anything? Do you, maybe, monitor the load on those windows servers?

Thanks

Replied with the details in a private comment.

Hi,

Do not use MySQL Server 8.0.29 please. You can test 8.0.29 router.

If I understood properly you moved the router to .28 and left Server at .30 and the crashes stopped. I cannot still reproduce crashes locally, please let me know if you move to .29 or .30 (router only not changing server) and the crashes start again.

Please send me the full config (router and server and connect string you are using from your app) so I can try to reproduce this again using configuration as similar as possible to yours

Thanks

Replied in a private comment, thanks!

Thanks for all the details Keith,

I hope we will manage to figure out what is making this crashes happen between .28 and .29. 

have a nice vacation

I'm available for more testing if needed. Would it help if I tried to reproduce it using sysbench to hopefully make it more repeatable?

Also, if there's a way for me to have a crash dump generated for mysqlrouter.exe when it dies, let me know and I'll try to capture that as well.

Thanks!

Posted by developer:
 
It would help to start the 8.0.30 router with the --core-file.

$ mysqlrouter -c {configfile} --core-file

continue with the testing until it crashes, and check for a *.dmp file (a minidump).

I tried this out but it appears mysqlrouter.exe on Windows doesn't have a --core-file option available. Is there an equivalent on Windows?

Posted by developer:
 
Hi Keith,

I believe the --core-file functionality has just become available on Windows in Router 8.0.31.  Could you give it a try?  I see Router 8.0.31 is already available on dev.mysql.com.

For your convenience, here's additional info:
https://dev.mysql.com/doc/mysql-router/8.0/en/mysql-router-conf-options.html#option_mysqlr...

MySQL Router 8.0.31 seems to have started successfully with the --core-file option. I will update this ticket when it crashes next with the core file.

Can confirm that MySQL Router 8.0.31 still crashes on Windows, but it did not generate a crash dump even though it was running with the --core-file option. I searched the entire drive for *.dmp files, and there aren't any for MySQL Router.

Windows can be configured to generate crash dumps,  maybe a full crash dump will help if repeatability is an issue.

https://learn.microsoft.com/en-us/windows/win32/wer/collecting-user-mode-dumps

mysqlslap with a high --iterations and --concurrency with a low --number-of-queries  will stress test the connections code..

I completely missed your last reply on this, apologies for the delay! I will see if I can get a user-mode dump for the crash.

I've been running 8.0.32 on the affected machine since Thursday and haven't had any crashes yet. I will monitor for another week and report back.

MySQL Router 8.0.32 has been running on the affected machines for over a week now with no crashing issues, so I think it's safe to mark this as resolved :)

Posted by developer:
 
Confirmed to be fixed in at least 8.0.32 by reporter.

Posted by developer:
 
Added the following note to the MySQL Router 8.0.33 release notes:
	
  Under certain circumstances, on Microsoft Windows platforms, MySQL Router crashed repeatedly.