MySQL Bugs: #113393: MySQL Server locks up randomly

Bug #113393	MySQL Server locks up randomly
Submitted:	11 Dec 2023 14:57	Modified:	14 Jun 2024 9:27
Reporter:	Paul Smith	Email Updates:
Status:	Not a Bug	Impact on me:	None
Category:	MySQL Server	Severity:	S2 (Serious)
Version:	8.0.32	OS:	CentOS (7)
Assigned to:		CPU Architecture:	Any

Description:
Over the past few years we've experienced several instances where MySQL completely locks up. We first noticed this issue in MySQL 5.7.23 and we're still seeing it after upgrading to MySQL 8.0.32. MySQL 5.7.19 does not appear to have this problem.

The problem occurs randomly after the server has been running for around 1-4 weeks (sometimes only a few days).
It only happens on our production environment which has quite a high write heavy load.
Higher load seems to make it happen sooner.
MySQL still accepts the TCP connections (until they're all in use) but doesn't seem to respond to any of them.
It appears that all connections already in progress stop responding.
The one connection reserved for root access also doesn't respond but I don't think it even gets as far as authenticating.

When in this state, running "systemctl restart mysqld" also hangs indefinitely and I have to send a SIGKILL to one of the MySQL processes.
It doesn't seem to matter which process I kill, the rest then all close and the restart command goes through as expected.

We use replication. I don't think there's an issue reading/processing the relay logs but I can't rule out a bug writing to the binlogs which I believe has had memory leaks in older versions.
Using master-master replication, we see either server lock up.
Using master-slave replication, we have only seen the master lock up.
Of course, it could have nothing to do with replication and may be something that only affects direct queries.

All of our tables are InnoDB. We've seen the same issue on servers running 1 database with around 100 tables totalling less than 2GB and also servers that have 10,000 databases with around 10 tables each totalling over 1TB.

I'm sorry I can't narrow it down any further. I've spent months trying to get to the bottom of this with no luck.
I'd be happy to run tests (within reason as these are production servers) if you can think of anything that would help.

How to repeat:
Unknown

Hi Mr. Smith,

Thank you for your bug report.

However, we have to inform you that this is a forum only for the reports with a fully repeatable test cases, made in SQL statements, with other relevant informations, like configurations and similar, included when necessary.

Hence, this is not a report that we can process.

We believe that you have serious problems, but your problems look like they require a proper support.

If you wish us to give you URL's for the free and commercial support, just let us know.

Can't repeat.

Hi, We still haven't had any luck narrowing it down. Could you please provide the URLs for support?

HI Mr. Smith,

Thank you for your question.

We are also providing you with the info that you asked for.

For details on getting support for MySQL products see http://www.mysql.com/support/
You can also check our forums (free) at http://forums.mysql.com/

Thank you for your interest in MySQL.