Description:
First there seems to be several related bugs but none of them seem to have any actions being taken on them. SO....
Very INFREQUENTLY I get an exception about "Reading from the stream has failed". This is occurring while calling Open(); I would say that about 99% of the time it functions correctly. Meaning it is NOT a password or username problem. (Same are being used in all cases)
Environment: AWS RDS MySQL server.
Application: .Net app(Server) running on a Windows EC2 Instance.
The server receives connections from client apps and then spins off a task which connects to MySQL and makes a query and returns the result to the client app.
Once in a while I receive an exception (will post it below) and the connection fails. The problem is that from that time on, the server ALWAYS fails on connections AND worst of all brings the server to its knees performance wise running at 100% cpu processing. I have tested and verified that if I randomly throw exceptions at the same location the CPU utilization continues to perform at its normal 2%.
It appears that something is failing badly in the API and once it does, it is attempting to fix itself with continued calls? This case effects the entire machine. In some cases I have multiple server applications running on the same hardward. All applications start failing on the Open() call. HOWEVER other machines are not affected by the problem. Mysql Workbench from my desktop works fine and I can not see any problems with the DB's performance.
When I shut the server app down the machine comes back and I can restart the app and everything works fine(NOT A SOLUTION)
Here is the Exceptions (and inner Exceptions and callstacks)
Authentication to host 'ministatic.ccaugtcoz2r0.us-west-2.rds.amazonaws.com' for user 'miniidol' using method 'mysql_native_password' failed with message: Reading from the stream has failed.
at MySql.Data.MySqlClient.Authentication.MySqlAuthenticationPlugin.AuthenticationFailed(Exception ex)
at MySql.Data.MySqlClient.Authentication.MySqlAuthenticationPlugin.ReadPacket()
at MySql.Data.MySqlClient.Authentication.MySqlAuthenticationPlugin.Authenticate(Boolean reset)
at MySql.Data.MySqlClient.NativeDriver.Authenticate(String authMethod, Boolean reset)
at MySql.Data.MySqlClient.NativeDriver.Open()
at MySql.Data.MySqlClient.Driver.Open()
at MySql.Data.MySqlClient.Driver.Create(MySqlConnectionStringBuilder settings)
at MySql.Data.MySqlClient.MySqlConnection.Open()
at ServerCommon.Lib.PersistentData.SqlConnect..ctor(String database) in j:\Minitropolis\Source\ServerCommon\ServerCom
mon.PersistantData\Base\SQLConnect.cs:line 56
Authentication to host 'ministatic.ccaugtcoz2r0.us-west-2.rds.amazonaws.com' for user 'miniidol' using method 'mysql_native_password' failed with message: Reading from the stream has failed.
at ServerCommon.Lib.PersistentData.SqlConnect..ctor(String database) in j:\Minitropolis\Source\ServerCommon\ServerCommon.PersistantData\Base\SQLConnect.cs:line 65
at ServerCommon.Lib.PersistentData.PersistentDataConnection..ctor(Boolean write) in j:\Minitropolis\Source\ServerCommon\ServerCommon.PersistantData\Base\PersistentDataContext.cs:line 814
at RealtimeServer.Server.Lib.Citizen.LoadData(Boolean blocking) in j:\Minitropolis\Source\RealtimeServer\RealtimeServer.Server.Lib\RuntimeData\Citizen.cs:line 245
at RealtimeServer.Server.Lib.Citizen.OnUpdateLoopStarted() in j:\Minitropolis\Source\RealtimeServer\RealtimeServer.Server.Lib\RuntimeData\Citizen.cs:line 976
at RealtimeServer.Server.Lib.Messagable`1.StartUpdateLoop() in j:\Minitropolis\Source\RealtimeServer\RealtimeServer.Server.Lib\RuntimeData\Messagable.cs:line 69
at RealtimeServer.Server.Lib.Citizen.EnsureUpdateLoop() in j:\Minitropolis\Source\RealtimeServer\RealtimeServer.Server.Lib\RuntimeData\Citizen.cs:line 945
at RealtimeServer.Server.Lib.Citizen.Setup() in j:\Minitropolis\Source\RealtimeServer\RealtimeServer.Server.Lib\RuntimeData\Citizen.cs:line 184
at RealtimeServer.Server.Lib.PacketHandlers.ActualJoinHandler.ProcessJoin(ClientPacket packet, Connection connection)
in j:\Minitropolis\Source\RealtimeServer\RealtimeServer.Server.Lib\PacketHandlers\ActualJoinHandler.cs:line 109
Reading from the stream has failed.
at MySql.Data.MySqlClient.MySqlStream.LoadPacket()
at MySql.Data.MySqlClient.MySqlStream.ReadPacket()
at MySql.Data.MySqlClient.Authentication.MySqlAuthenticationPlugin.ReadPacket()
Attempted to read past the end of the stream.
at MySql.Data.MySqlClient.MySqlStream.ReadFully(Stream stream, Byte[] buffer, Int32 offset, Int32 count)
at MySql.Data.MySqlClient.MySqlStream.LoadPacket()
If there is any additional information I can provide Please let me know!
How to repeat:
Unfortunately I can not reproduce this bug! It happens infrequently enough that I normally don't care but now that I have about to release this to a production environment I can't have the server going down and in the rare case that this occurs I would have to restart the server app.
