MySQL Bugs: #103260: Unable to restart mysql server

Bug #103260	Unable to restart mysql server
Submitted:	9 Apr 2021 4:21	Modified:	20 Apr 2021 23:32
Reporter:	Eugene Chan	Email Updates:
Status:	Can't repeat	Impact on me:	None
Category:	MySQL Server	Severity:	S3 (Non-critical)
Version:	8.0.21	OS:	Red Hat (7.9)
Assigned to:		CPU Architecture:	Any

Description:
Have been unable to restart mysql server since at least 15th March though I cannot confirm when the exact date the server wasn't able to be restarted.

Looked through the yum log, the closest updates were installed on the 6th March but the only package I see that might be related is glibc was updated

Have checked the state of the MYI files using myisamchk 

Logs
------
The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you find out what is causing the crash.
2021-04-09T01:41:56.506079Z 0 [System] [MY-010116] [Server] /opt/rh/rh-mysql80/root/usr/libexec/mysqld (mysqld 8.0.21) starting as process 16488
2021-04-09T01:41:56.515830Z 1 [System] [MY-013576] [InnoDB] InnoDB initialization has started.
2021-04-09T01:41:56.634276Z 1 [ERROR] [MY-012671] [InnoDB] Encryption algorithm support missing: N
2021-04-09T01:41:56.634392Z 1 [ERROR] [MY-013183] [InnoDB] Assertion failure: log0recv.cc:3540:err == DB_SUCCESS thread 140407262598912
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: http://dev.mysql.com/doc/refman/8.0/en/forcing-innodb-recovery.html
InnoDB: about forcing recovery.
01:41:56 UTC - mysqld got signal 6 ;
Most likely, you have hit a bug, but this error can also be caused by malfunctioning hardware.
Thread pointer: 0x555cdf164bb0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 7fb31d02cc40 thread_stack 0x46000
/opt/rh/rh-mysql80/root/usr/libexec/mysqld(my_print_stacktrace(unsigned char const*, unsigned long)+0x3d) [0x555cdaff865d]
/opt/rh/rh-mysql80/root/usr/libexec/mysqld(handle_fatal_signal+0x2fb) [0x555cda01347b]
/lib64/libpthread.so.0(+0xf630) [0x7fb32c547630]
/lib64/libc.so.6(gsignal+0x37) [0x7fb32a1a33d7]
/lib64/libc.so.6(abort+0x148) [0x7fb32a1a4ac8]
/opt/rh/rh-mysql80/root/usr/libexec/mysqld(ut_dbg_assertion_failed(char const*, char const*, unsigned long)+0x2de) [0x555cdb2da5de]
/opt/rh/rh-mysql80/root/usr/libexec/mysqld(+0x2198da8) [0x555cdb18bda8]
/opt/rh/rh-mysql80/root/usr/libexec/mysqld(recv_recovery_from_checkpoint_start(log_t&, unsigned long)+0x609) [0x555cdb1959e9]
/opt/rh/rh-mysql80/root/usr/libexec/mysqld(srv_start(bool)+0x2106) [0x555cdb283f26]
/opt/rh/rh-mysql80/root/usr/libexec/mysqld(+0x20fcce6) [0x555cdb0efce6]
/opt/rh/rh-mysql80/root/usr/libexec/mysqld(dd::bootstrap::DDSE_dict_init(THD*, dict_init_mode_t, unsigned int)+0x90) [0x555cdad4d500]
/opt/rh/rh-mysql80/root/usr/libexec/mysqld(dd::upgrade_57::do_pre_checks_and_initialize_dd(THD*)+0x5bc) [0x555cdafc304c]
/opt/rh/rh-mysql80/root/usr/libexec/mysqld(+0x10e4aa5) [0x555cda0d7aa5]
/opt/rh/rh-mysql80/root/usr/libexec/mysqld(+0x2552ac1) [0x555cdb545ac1]
/lib64/libpthread.so.0(+0x7ea5) [0x7fb32c53fea5]
/lib64/libc.so.6(clone+0x6d) [0x7fb32a26b9fd]

Trying to get some variables.
Some pointers may be invalid and cause the dump to abort.
Query (0): is an invalid pointer
Connection ID (thread ID): 1
Status: NOT_KILLED

The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you find out what is causing the crash.

How to repeat:
Happens every time I try and restart the server.

Hi Mr. Chan,

Thank you for your bug report.

The problems that you are experiencing are not due to the bad MyISAM files, but are due to the botched upgrade from 5.7 to 8.0.

Please, start from 5.7, check all tables and metadata and follow the upgrade process carefully. This is all described in our Reference Manual. Also, please use 8.0.23.

I don't think this instance ever had 5.7 installed but I will try to downgrade and re-upgrade the server.

Hi Mr. Chan,

The stacktrace clearly shows that it was mysql-5.7 installation. You should follow our Reference Manual on the upgrade process and pay attention to the details about which 5.7 release did you use before switching to 8.0.

So resolved this issue by backing up and then clearing out the data directory. 

Restarted the server and recreated the users, database and tables from a dump of the development instance. I then copied the MYI and MYD files from the backup table by table, restarting the server each time to make sure each table did not break the server.

Each table was copied without issue and the server is now up and running.