MySQL Bugs: #107665: mysql router crash

Bug #107665	mysql router crash
Submitted:	27 Jun 2022 6:02	Modified:	14 Oct 2022 6:13
Reporter:	WANG FENG	Email Updates:
Status:	Duplicate	Impact on me:	None
Category:	MySQL Router	Severity:	S1 (Critical)
Version:	8.0.29	OS:	CentOS (7.7.1908)
Assigned to:		CPU Architecture:	x86

Description:
I had three nodes innodbCluster. And Mysql router was deployed on every three nodes. 
Mysql version was 8.0.26-el7-x86_64.
Mysql router version mysql-router-8.0.29-el7-x86_64.

Today,5:01:45AM, all mysqlRouter instance crashed about the same time.
I got nothing message exception following 
"
[root@WCM-MYSQLDB-03-LIUSJ log]# grep router messages
Jun 26 05:01:44 WCM-MYSQLDB-03-LIUSJ kernel: io-2[20582]: segfault at 0 ip 00007ff1c15b04a5 sp 00007ff1bbffeb50 error 4 in libmysqlrouter_routing.so.1[7ff1c155b000+63000]

[root@WCM-MYSQLDB-02-LIUSJ log]# grep router messages
Jun 26 05:01:45 WCM-MYSQLDB-02-LIUSJ kernel: io-2[22372]: segfault at 0 ip 00007fbb8b4884a5 sp 00007fbb8a37fb50 error 4 in libmysqlrouter_routing.so.1[7fbb8b433000+63000]

[root@wcm-mysqldb-01-liusj log]# grep router messages
Jun 26 05:01:45 WCM-MYSQLDB-01-LIUSJ kernel: io-2[6657]: segfault at 0 ip 00007f35735f94a5 sp 00007f35724f0b60 error 4 in libmysqlrouter_routing.so.1[7f35735a4000+63000]

"

How to repeat:
have no idea.

Suggested fix:
have no idea.

Hi,

Thank you for the report but without a way to reproduce this there is nothing we can do.

Any advice about  this message "segfault at 0 ip 00007ff1c15b04a5 sp 00007ff1bbffeb50 error 4 in libmysqlrouter_routing.so.1[7ff1c155b000+63000]"? Or you can give me some advices to research this ? Thanks.

Hi,

IMHO segfault is always a bug, and since it is our library that is segfaulting it is our bug so we should fix it. The problem is - without ability to reproduce it chances of fixing it are next to none. There is always a chance of external influence (e.g. bar hardware) but that is another story. 

We need to know when it happens so if you can somehow figure out how / when you are getting this segfault - what is the order of operations, and get to at least some vicinity of reproducible test case I can take over and try to make a reproducible test case. But with just the segfault info not much we can do.

The other thing would be to run a debug build so we can extract more data from the crash when it happens.

Thanks

Hi,
   Thanks for your advices. It shoud be a reasonable approach for me to running a debug build at this point. Can any where i get a mysqlrouter debug build release? Thanks again.

Hi,
   Thanks for your advices. It shoud be a reasonable approach for me to running a debug build at this point. Can any where i get a mysqlrouter debug build release? Thanks again.

Hi,did the following message has any help for this?

Aug 16 15:20:42 bom-mysql3-app start.sh: stack_bottom = 0 thread_stack 0x0
Aug 16 15:20:42 bom-mysql3-app start.sh: /usr/local/mysql-router-8.0.29-el7-x86_64/bin/../lib/mysqlrouter/private/libmysqlharness.so.1(my_print_stacktrace(unsigned char const*, unsigned long)+0x2e) [0x7f3b5c9c24be]
Aug 16 15:20:42 bom-mysql3-app start.sh: /lib64/libpthread.so.0(+0xf5f0) [0x7f3b5c5f65f0]
Aug 16 15:20:42 bom-mysql3-app start.sh: /usr/local/mysql-router-8.0.29-el7-x86_64/lib/mysqlrouter/private/libmysqlrouter_routing.so.1(MysqlRoutingXConnection::async_send_server(MysqlRoutingXConnection::Function)+0x35) [0x7f3b59d124a5]
Aug 16 15:20:42 bom-mysql3-app start.sh: /usr/local/mysql-router-8.0.29-el7-x86_64/lib/mysqlrouter/private/libmysqlrouter_routing.so.1(MysqlRoutingXConnection::client_recv_cmd()+0x248) [0x7f3b59d1a948]
Aug 16 15:20:42 bom-mysql3-app start.sh: /usr/local/mysql-router-8.0.29-el7-x86_64/bin/../lib/mysqlrouter/private/libmysqlrouter_io_component.so.1(IoThread::operator()()+0x2b1) [0x7f3b5ca06b01]
Aug 16 15:20:42 bom-mysql3-app start.sh: /usr/local/mysql-router-8.0.29-el7-x86_64/bin/../lib/mysqlrouter/private/libmysqlharness.so.1(+0xe7924) [0x7f3b5c9cc924]
Aug 16 15:20:42 bom-mysql3-app start.sh: /lib64/libpthread.so.0(+0x7e65) [0x7f3b5c5eee65]
Aug 16 15:20:42 bom-mysql3-app start.sh: /lib64/libc.so.6(clone+0x6d) [0x7f3b5a53688d]
Aug 16 15:20:42 bom-mysql3-app kernel: io-3[66291]: segfault at 0 ip 00007f3b59d124a5 sp 00007f3b4bffeb60 error 4 in libmysqlrouter_routing.so.1[7f3b59cbd000+63000]
Aug 16 15:20:42 bom-mysql3-app systemd: mysql_router.service: main process exited, code=killed, status=11/SEGV
Aug 16 15:20:42 bom-mysql3-app stop.sh: /igwfmc/mysql/router/stop.sh: line 3: kill: (66287) - No such process
Aug 16 15:20:42 bom-mysql3-app systemd: mysql_router.service: control process exited, code=exited status=1
Aug 16 15:20:42 bom-mysql3-app systemd: Unit mysql_router.service entered failed state.
Aug 16 15:20:43 bom-mysql3-app systemd: Stopped MySQL Router.
Aug 16 15:20:43 bom-mysql3-app start.sh: stopping to log to the console. Continuing to log to filelog
Aug 16 15:29:44 bom-mysql3-app start.sh: stack_bottom = 0 thread_stack 0x0
Aug 16 15:29:44 bom-mysql3-app start.sh: /usr/local/mysql-router-8.0.29-el7-x86_64/bin/../lib/mysqlrouter/private/libmysqlharness.so.1(my_print_stacktrace(unsigned char const*, unsigned long)+0x2e) [0x7fa29fa9d4be]
Aug 16 15:29:44 bom-mysql3-app start.sh: /lib64/libpthread.so.0(+0xf5f0) [0x7fa29f6d15f0]
Aug 16 15:29:44 bom-mysql3-app start.sh: /usr/local/mysql-router-8.0.29-el7-x86_64/lib/mysqlrouter/private/libmysqlrouter_routing.so.1(MysqlRoutingXConnection::async_send_server(MysqlRoutingXConnection::Function)+0x35) [0x7fa29cded4a5]
Aug 16 15:29:44 bom-mysql3-app start.sh: /usr/local/mysql-router-8.0.29-el7-x86_64/lib/mysqlrouter/private/libmysqlrouter_routing.so.1(MysqlRoutingXConnection::client_recv_cmd()+0x248) [0x7fa29cdf5948]
Aug 16 15:29:44 bom-mysql3-app start.sh: /usr/local/mysql-router-8.0.29-el7-x86_64/bin/../lib/mysqlrouter/private/libmysqlrouter_io_component.so.1(IoThread::operator()()+0x2b1) [0x7fa29fae1b01]
Aug 16 15:29:44 bom-mysql3-app start.sh: /usr/local/mysql-router-8.0.29-el7-x86_64/bin/../lib/mysqlrouter/private/libmysqlharness.so.1(+0xe7924) [0x7fa29faa7924]
Aug 16 15:29:44 bom-mysql3-app start.sh: /lib64/libpthread.so.0(+0x7e65) [0x7fa29f6c9e65]
Aug 16 15:29:44 bom-mysql3-app start.sh: /lib64/libc.so.6(clone+0x6d) [0x7fa29d61188d]
Aug 16 15:29:44 bom-mysql3-app kernel: io-0[10185]: segfault at 0 ip 00007fa29cded4a5 sp 00007fa29cd28b50 error 4 in libmysqlrouter_routing.so.1[7fa29cd98000+63000]
Aug 16 15:29:44 bom-mysql3-app systemd: mysql_router.service: main process exited, code=killed, status=11/SEGV
Aug 16 15:29:44 bom-mysql3-app stop.sh: /igwfmc/mysql/router/stop.sh: line 3: kill: (10184) - No such process
Aug 16 15:29:44 bom-mysql3-app systemd: mysql_router.service: control process exited, code=exited status=1
Aug 16 15:29:44 bom-mysql3-app systemd: Unit mysql_router.service entered failed state.
Aug 16 15:29:44 bom-mysql3-app systemd: mysql_router.service failed.
Aug 16 15:29:44 bom-mysql3-app systemd: mysql_router.service holdoff time over, scheduling restart.
Aug 16 15:29:44 bom-mysql3-app systemd: Stopped MySQL Router.
Aug 16 15:29:44 bom-mysql3-app systemd: Started Session c10 of user root.
Aug 16 15:31:01 bom-mysql3-app systemd: Started Session 80452 of user root.
Aug 16 15:31:37 bom-mysql3-app start.sh: Application got fatal signal: 11
Aug 16 15:31:37 bom-mysql3-app start.sh: stack_bottom = 0 thread_stack 0x0
Aug 16 15:31:37 bom-mysql3-app start.sh: /usr/local/mysql-router-8.0.29-el7-x86_64/bin/../lib/mysqlrouter/private/libmysqlharness.so.1(my_print_stacktrace(unsigned char const*, unsigned long)+0x2e) [0x7f16ac7584be]
Aug 16 15:31:37 bom-mysql3-app start.sh: /lib64/libpthread.so.0(+0xf5f0) [0x7f16ac38c5f0]
Aug 16 15:31:37 bom-mysql3-app start.sh: /usr/local/mysql-router-8.0.29-el7-x86_64/lib/mysqlrouter/private/libmysqlrouter_routing.so.1(MysqlRoutingXConnection::async_send_server(MysqlRoutingXConnection::Function)+0x35) [0x7f16a9aa84a5]
Aug 16 15:31:37 bom-mysql3-app start.sh: /usr/local/mysql-router-8.0.29-el7-x86_64/lib/mysqlrouter/private/libmysqlrouter_routing.so.1(MysqlRoutingXConnection::client_recv_cmd()+0x248) [0x7f16a9ab0948]
Aug 16 15:31:37 bom-mysql3-app start.sh: /usr/local/mysql-router-8.0.29-el7-x86_64/bin/../lib/mysqlrouter/private/libmysqlrouter_io_component.so.1(IoThread::operator()()+0x2b1) [0x7f16ac79cb01]
Aug 16 15:31:37 bom-mysql3-app start.sh: /usr/local/mysql-router-8.0.29-el7-x86_64/bin/../lib/mysqlrouter/private/libmysqlharness.so.1(+0xe7924) [0x7f16ac762924]
Aug 16 15:31:37 bom-mysql3-app start.sh: /lib64/libpthread.so.0(+0x7e65) [0x7f16ac384e65]
Aug 16 15:31:37 bom-mysql3-app start.sh: /lib64/libc.so.6(clone+0x6d) [0x7f16aa2cc88d]
Aug 16 15:31:37 bom-mysql3-app kernel: io-3[11183]: segfault at 0 ip 00007f16a9aa84a5 sp 00007f16a3ffeb60 error 4 in libmysqlrouter_routing.so.1[7f16a9a53000+63000]
Aug 16 15:31:37 bom-mysql3-app systemd: mysql_router.service: main process exited, code=killed, status=11/SEGV
Aug 16 15:31:37 bom-mysql3-app stop.sh: /igwfmc/mysql/router/stop.sh: line 3: kill: (11179) - No such process
Aug 16 15:31:37 bom-mysql3-app systemd: mysql_router.service: control process exited, code=exited status=1
Aug 16 15:31:37 bom-mysql3-app systemd: Unit mysql_router.service entered failed state.
Aug 16 15:31:37 bom-mysql3-app systemd: mysql_router.service failed.
Aug 16 15:31:38 bom-mysql3-app systemd: Starting MySQL Router...

Hi,

not much :( unfortunately.

Please configure process to generate core, then load the code using gdb and execute backtrace, that would be a good starting point. Or if you can upload the core to the bug so we can analyze it. Normally we would do this analysis but I cannot reproduce the problem.

Thanks

Hi,
  Thanks gods, i am so luck. 
  I get the core dump file after crash this time.
  I had upload bug data to sftp.oracle.com:/support/incoming with filename mysql-bug-data-107665.zip. Maybe you can get some usefull information. thanks.

Hi, Does anybody deal with this?

sftp> get /support/incoming/mysql-bug-data-107665.zip
File "/support/incoming/mysql-bug-data-107665.zip" not found.

Hi,
I need to full proper filename you uploaded to the ftp in order to get it

thanks

Hi,A problem happend when i uploading mysql-bug-data-107665.zip , May be this file had exists. So i uploading another file mysql-bug-data-107665-1.zip. Detail
follows.

xxx\wf\problem\mysqlrouter>sftp -oPort=2021 -oUser=wangf@igwfmc.com sftp.oracle.com:/support/incoming
*******************************************************************************
Oracle SFTP Server

The use of this SFTP server is governed by United States Export Administration
Regulations. Details of the U.S. Commercial Encryption Export Controls can be
found at the Bureau of Industry and Security web site.
All Oracle products are subject to U.S. Export Laws.
Diversion contrary to U.S. law is prohibited

http://www.oracle.com/us/products/export/export-regulations-345813.html

Oracle Employees:
Under no circumstances are Oracle Employees authorized to download software for
the purpose of distributing it to customers. Oracle products are available to
employees for internal use or demonstration purposes only. Customers may
download Oracle software through the Oracle Technology Network or through
Oracle's e-Delivery site. Failure to comply with these requirements may result
in disciplinary action.
****************************************************************************
password
Enter password for wangf@igwfmc.com
Password:
Connected to sftp.oracle.com.
Changing to: /support/incoming
sftp> put mysql-bug-data-107665.zip
Uploading mysql-bug-data-107665.zip to //support/incoming/mysql-bug-data-107665.zip
remote open("//support/incoming/mysql-bug-data-107665.zip"): Failure
sftp> put mysql-bug-data-107665-1.zip
Uploading mysql-bug-data-107665-1.zip to //support/incoming/mysql-bug-data-107665-1.zip
mysql-bug-data-107665-1.zip 100% 7417KB 1.5MB/s 00:05
Calculating SHA256 Checksum [ | ]sftp>
SHA-256 Checksum: cf583ddee4295d481b0075db1d47a308695660ee0858124e494052cffa085ceb

Skipping Virus Scan for mysql-bug-data-107665-1.zip due to already scanned..

sftp> ls
sftp>

Thanks, I have the file now

Posted by developer:
 
There is another bug in the system, which looks very similar, so I closed it off as a duplicate:
  BUG#34654228

Note that it provides another core file though, perhaps useful:
  href=http://bugs.mysql.com/file.php?id=31762

Hi,
  Did you mean this "https://bugs.mysql.com/bug.php?id=108609"?
  Is there any solution that i can take? Thanks.

Posted by developer:
 
The reported bug is looks like the same as Bug#34597056.

A fix as been applied and will be part of the next release.

Hi，
   Is there a workaround solution for now? Thank you.

If you don't need the x-protocol support, you can disable its support in the mysqlrouter.cnf by removing the [routing:...] sections which contain "protocol=x".

By default, the bootstrap creates 4 [routing:...] sections:

- 2 for "protocol=classic" <- keep those
- 2 for "protocol=x"       <- remove those

Ok. Thank you.