MySQL Bugs: #118565: MySQL Router 'Application got fatal signal: 11' segfault with connection pooling enabled

Bug #118565	MySQL Router 'Application got fatal signal: 11' segfault with connection pooling enabled
Submitted:	30 Jun 16:42	Modified:	10 Jul 15:48
Reporter:	Humberto Adolfo Leal Betancur	Email Updates:
Status:	Verified	Impact on me:	None
Category:	MySQL Router	Severity:	S2 (Serious)
Version:	Latest 9.3.0	OS:	Any
Assigned to:		CPU Architecture:	Any
Tags:	Fatal, router, segfault

Description:
We have observed periodic crashes (Segmentation fault) on multiple instances of MySQL Router running on a production workload. After enabling and analysing core dumps (see attached file) it appears that the error arises in the connection pooling logic in combination with TLS/SSL handling.

We run MySQL Router in front of MySQL instances running with InnoDB group replication in a Kubernetes environment. We have seen the issue happening with different versions of MySQL Router (9.3.0 as well as 8.0.40).
Configuration used on the MySQL Router instance is the one generated during the bootstrap process, no other config values are modified (attached file for reference).

After reviewing the stack trace and files that might be involved with the issue, we created other environments and tried different settings in an attempt to prevent the issue from happening. Setting `DEFAULT.max_idle_server_connections=0` in order to disable connection pooling seems to have resolved the issue for us. Ideally we still would like to make use of connection pooling.

Based on the observations described above, we strongly believe that this is a bug in the MySQL Router connection pooling logic.

How to repeat:
We have not found a clear, deterministic path to reproduce the issue. We came up with a script that tries different approaches in terms of spawning multiple connections to test connection pooling and alternate connections with SSL enabled/disabled. The issue was observed with this setup at least once or twice a day. On our production workloads, the issue happened more frequently on instances with higher load patterns.

We have attached the PHP script used for for the reproduction. It was deployed as a client pod in our Kubernetes environment. Dockerhub image is also available via `champgoblem/mysql-segfault:latest`. The script can be started with following command: `php ./main.php <HOST> 3306 <USER> <PASSWORD> <DB>`.

Suggested fix:
Error likely occurs in connection_pool.h (https://dev.mysql.com/doc/dev/mysql-server/9.3.0/connection__pool_8h_source.html) but we lack context to properly assess the exact root cause.

Stack trace

Attachment: stack_trace.txt (text/plain), 6.06 KiB.

router configuration generated by bootstrap process

Attachment: router-config (application/octet-stream, text), 2.23 KiB.

Edit: had trouble uploading the php script, it's available here instead https://github.com/hleal18/php-conn-pool-script/blob/main/script.php/

Hi,

Thank you for the report. I was unable to reproduce it but I'll check if router team can extract needed info from stack trace you provided. Thanks.

Looks like we're affected by this, too.

Crashes like once a day.

Also Kubernetes, also still happening with 9.3.0

Let me know if there's anything I can provide to help track this down.

Application got fatal signal: 11                                                                                                                                                                                                                                                                                                                  │ stack_bottom = 0 thread_stack 0x0                                                                                                                                                                                                                                                                                                                 │  #0 0x7168eefad7c0 <unknown>                                                                                                                                                                                                                                                                                                                      │  #1 0x7168edcba72f <unknown>                                                                                                                                                                                                                                                                                                                      │  #2 0x7168ec8e8374 <unknown>                                                                                                                                                                                                                                                                                                                      │  #3 0x7168ec8e8fa4 <unknown>                                                                                                                                                                                                                                                                                                                      │  #4 0x7168ec8b9ce8 <unknown>                                                                                                                                                                                                                                                                                                                      │  #5 0x7168ec8ba2b8 <unknown>                                                                                                                                                                                                                                                                                                                      │  #6 0x7168ec894a11 <unknown>                                                                                                                                                                                                                                                                                                                      │  #7 0x7168ef0cb841 <unknown>                                                                                                                                                                                                                                                                                                                      │  #8 0x7168ef0ce4af <unknown>                                                                                                                                                                                                                                                                                                                      │  #9 0x7168ee054ad3 <unknown>                                                                                                                                                                                                                                                                                                                      │  #10 0x7168edd057f1 <unknown>                                                                                                                                                                                                                                                                                                                     │  #11 0x7168edd8a87f <unknown>                                                                                                                                                                                                                                                                                                                     │  #12 0xffffffffffffffff <unknown>

We're running a few hundred million queries through this mysql router instance each day, in case this makes any difference.

Hi Johnathan,

Adding as much info on your setup will surely help. Also any logs you can share will help. We do not have clear and repeatable way of reproducing the problem.

Is there a docker image that produces more debug information?

Host system:

# uname -a
Linux node1 6.8.0-63-generic #66-Ubuntu SMP PREEMPT_DYNAMIC Fri Jun 13 20:25:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux

# cat /etc/os-release 
PRETTY_NAME="Ubuntu 24.04.2 LTS"
NAME="Ubuntu"
VERSION_ID="24.04"
VERSION="24.04.2 LTS (Noble Numbat)"

# cat /proc/cpuinfo  | head
processor       : 0
vendor_id       : AuthenticAMD
cpu family      : 25
model           : 97
model name      : AMD EPYC 4344P 8-Core Processor
stepping        : 2
microcode       : 0xa601209

But we've been seeing the same issue on Xeon nodes as well

Moved some load off the router instance and it took 3 days for it to crash.

Maybe this is some race condition with a very small probability?