MySQL Bugs: #99835: mysql system table corrupted randomly for several times

Bug #99835	mysql system table corrupted randomly for several times
Submitted:	10 Jun 2020 8:07	Modified:	12 Jun 2020 1:11
Reporter:	wu winnie	Email Updates:
Status:	Unsupported	Impact on me:	None
Category:	MySQL Server: Windows	Severity:	S3 (Non-critical)
Version:	5.6.30	OS:	Windows
Assigned to:	MySQL Verification Team	CPU Architecture:	x86
Tags:	corrupted

Description:
Our product is running on WINDOWS OS, and it contains a MySQL server.
when our product runs a script, it needs to start MySQL server. when it detects the script finishes, it will shutdown MySQL server.
our customer has a windows task schedule and set the script runs every other 30 minutes, After 1 or 2 days, he found the script ran failed, which was caused by MySQL server can't be started, this issue has happened several times this month, every time we have to resolve it by recreate MySQL data folder.

the error shows in MySQL err file:
****************************************************************************
2020-06-05 03:31:11 10744 [ERROR] C:\Program Files (x86)\Quest Software\Toad Data Point 5.1\datahub\bin\mysqld.exe: Incorrect key file for table '.\mysql\procs_priv'; try to repair it
2020-06-05 03:31:11 10744 [ERROR] Fatal error: Can't open and lock privilege tables: Incorrect key file for table '.\mysql\procs_priv'; try to repair it
****************************************************************************

I know this issue can be fixed by:
mysqlcheck.exe --user=root --port=xxxx --auto-repair --use-frm --databases MySQL
note I found I have to use --use-frm to repair this broken table.

But I was wondering why this happen.
1. he has enough free disk space for MySQL data
2. before this issue happen, I have checked MySQL server was shutdown normally previously

There's a very important clue:
We have deployed a self-designed MySQL plugin (storage engine). Every time after MySQL have started normally, we'll execute a query against this self-signed storage engine table. normally we will see following error message in MySQL err log file:

2020-06-02 05:01:05.646 [10980] [root] [4] companyKey.cpp:325 [ERROR] Cannot launch process: myTool.exe

But I found randomly this error message won't show, after MySQL server shutdown normally, I tried to start MySQL server, it failed to start and report certain system table is corrupted.

Then I go to check the corrupted table, and found the error message was written into procs_priv.MYI file.

I am quite confused, seems the MYI table was broken because the error message was incorrectly written into that file. I guess it's MySQL server bug.

How to repeat:
happen randomly. note the broken tables are not stable, sometimes MySQL.db, sometimes MySQL.procs_priv

Suggested fix:
I can upload the broken table files if you need, you'll see clear text of the error message.

frm file

Attachment: procs_priv.frm (application/octet-stream, text), 8.67 KiB.

broken MYI file

Attachment: procs_priv.myi (application/octet-stream, text), 8.00 KiB.

Hi,

This looks like not related to MySQL at all. 
The MYI file is corrupted with

2020-06-05 02:01:12.686 [10812] [root] [3] QuestKey.cpp:325 [ERROR] Cannot launch process: qklclw.exe

Neither QuestKey.cpp nor qklclw.exe are related to MySQL server so I believe you have some external process corrupting the file.

all best
Bogdan

The exe this error is complaining is failing to start seems to be part of Toad Data Point ( https://www.toadworld.com/products/toad-data-point ) and I'm not sure what's trying to start it, but possibly your storage engine, so looks like bug was introduced with your code.

all best
Bogdan

Hi,
  Thanks for your response. 
  yes, our product is TDP. and we have a self-designed storage engine in Mysql server. the error is thrown by our storage engine. 
  My question is: 
  1. Usually, the error message is printed in MySQL error log, but why was it written into MySQL system tables randomly? Note this issue happens randomly, not always.
  2. How could our own storage engine break the MYSQL's system tables?

note even the error message is written into MySQL system table, MySQL seems could work well. but next time when MySQL tries to start, it will report their system table is broken.

Hi,

> 1. Usually, the error message is printed in MySQL error log,
> but why was it written into MySQL system tables randomly?
> Note this issue happens randomly, not always.

Really nothing I can answer without looking at your code.

>  2. How could our own storage engine break the MYSQL's system tables?

Your code is operating from inside mysqld, there's nothing we can do to prevent you from writing into that file nor to write all over mysqld ram. A simple roaming pointer and you can write over output buffer.. 

all best
Bogdan

p.s. I'm sure you can get better help by opening a support ticket than bug report as this def. is not a MySQL bug; but in support ticket you can share code and get direct analysis of the problematic code.

ok, thanks. I'll check my source code and open a support ticket.
by the way, it's unnormal for a MySQL system table MYI file contains an error message, correct?

Hi,

> ok, thanks. I'll check my source code and open a support ticket.

Please do. Start with the file mentioned in the error message :) and with the file logging errors.

> by the way, it's unnormal for a MySQL system table MYI file contains an error message, correct?

Of course, that should never happen. But how that string eded up in the index file is impossible to determine without analysis of your code and that's code is I assume proprietary and not something you would like to have in public bug report hence my suggestion you open a support ticket.

all best
Bogdan

thanks very much. I will do some investigation and I think you can close this item now.