MySQL Bugs: #111177: Mysqlrouter crash "Application got fatal signal: 11"

Bug #111177	Mysqlrouter crash "Application got fatal signal: 11"
Submitted:	29 May 2023 3:03	Modified:	7 Nov 13:06
Reporter:	Duong Mai	Email Updates:
Status:	Verified	Impact on me:	None
Category:	MySQL Router	Severity:	S2 (Serious)
Version:	8.0.32, 8.0.33	OS:	Debian
Assigned to:		CPU Architecture:	Any

Description:
Hi
We are currently using mysqlrouter 8.0.32 in container (official images) with mysql innodb cluster (also in container using official images)
Mysql router containers were crashed some times with this backtrace:
```
Application got fatal signal: 11
stack_bottom = 0 thread_stack 0x0
/usr/lib64/mysqlrouter/private/libmysqlharness.so.1(my_print_stacktrace(unsigned char const*, unsigned long)+0x41) [0x7f147816bd41]
/usr/lib64/mysqlrouter/private/libmysqlharness.so.1(+0xd87f1) [0x7f14781567f1]
/lib64/libpthread.so.0(+0x12d00) [0x7f1477d8cd00]
/usr/lib64/mysqlrouter/private/libmysqlrouter_routing.so.1(MysqlRoutingClassicConnection::get_server_address[abi:cxx11]() const+0x42) [0x7f1474a85882]
/usr/lib64/mysqlrouter/private/libmysqlrouter_routing.so.1(void concurrent_map<MySQLRoutingConnectionBase*, std::shared_ptr<MySQLRoutingConnectionBase>, std::hash<MySQLRoutingConnectionBase*> >::for_each<ConnectionContainer::get_all_connections_info()::{lambda(std::pair<MySQLRoutingConnectionBase* const, std::shared_ptr<MySQLRoutingConnectionBase> > const&)#1}>(ConnectionContainer::get_all_connections_info()::{lambda(std::pair<MySQLRoutingConnectionBase* const, std::shared_ptr<MySQLRoutingConnectionBase> >/usr/lib64/mysqlrouter/private/libmysqlrouter_routing.so.1(MySQLRouting::get_connections()+0x38) [0x7f1474a5e6e8]
/usr/lib64/mysqlrouter/private/libmysqlrouter_routing.so.1(MySQLRoutingAPI::get_connections() const+0x29) [0x7f1474a807e9]
/usr/lib64/mysqlrouter/rest_routing.so(RestRoutingConnections::on_handle_request(HttpRequest&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&)+0x1de) [0x7f1474b974ae]
/usr/lib64/mysqlrouter/rest_api.so(RestApi::handle_paths(HttpRequest&)+0x729) [0x7f1474c14379]
/usr/lib64/mysqlrouter/http_server.so(HttpRequestRouter::route(HttpRequest)+0x16e) [0x7f1474c8cb2e]
/usr/lib64/mysqlrouter/http_server.so(+0x37e0c) [0x7f1474c8ce0c]
/usr/lib64/mysqlrouter/private/libmysqlrouter_http.so.1(+0xd6b5) [0x7f14780056b5]
/usr/lib64/mysqlrouter/private/libmysqlrouter_http.so.1(+0x3076c) [0x7f147802876c]
/usr/lib64/mysqlrouter/private/libmysqlrouter_http.so.1(+0x18b75) [0x7f1478010b75]
/usr/lib64/mysqlrouter/private/libmysqlrouter_http.so.1(+0x34e5d) [0x7f147802ce5d]
/usr/lib64/mysqlrouter/private/libmysqlrouter_http.so.1(+0x34e86) [0x7f147802ce86]
/usr/lib64/mysqlrouter/private/libmysqlrouter_http.so.1(+0x1ff42) [0x7f1478017f42]
/usr/lib64/mysqlrouter/private/libmysqlrouter_http.so.1(+0x20767) [0x7f1478018767]
/usr/lib64/mysqlrouter/http_server.so(HttpRequestThread::wait_and_dispatch()+0x35) [0x7f1474c7b5d5]
/lib64/libstdc++.so.6(+0xc2ba3) [0x7f1476514ba3]
/lib64/libpthread.so.0(+0x81da) [0x7f1477d821da]
/lib64/libc.so.6(clone+0x43) [0x7f1475b2ae73]
```

How to repeat:
This bug was happened randomly

we're also using a custom mysql router exporter to query REST API (60s interval) and translate to exporter format

Hi,

I cannot reproduce this. Can you test 8.0.33 ?

I also want to use 8.0.33 but I can not find 8.0.33 image on https://hub.docker.com/r/mysql/mysql-router/tags (newest is 8.0.32)
Where could I find newest image?

Please use Oracle container registry. You are using images that are not maintained by Oracle. We suggest you switch to Oracle repo

https://container-registry.oracle.com/ords/f?p=113:1:5838602727651:::1:P1_BUSINESS_AREA:6&...

docker pull container-registry.oracle.com/mysql/community-server:8.0.33

You can see that 8.0.33 is available since 5 weeks ago:

https://container-registry.oracle.com/ords/f?p=113:4:10808482767135:::RP,4:P4_REPOSITORY,A...

Okay, I will try 8.0.33 and report result later
Thanks

Hi, I tested with mysqlrouter 8.0.33 and still get a bug after 10 days. It happen randomly, not affect to all router instance 
Do you have any clues if look up stack strace ???

Application got fatal signal: 11
stack_bottom = 0 thread_stack 0x0
/usr/lib64/mysqlrouter/private/libmysqlharness.so.1(my_print_stacktrace(unsigned char const*, unsigned long)+0x41) [0x7ff388a10d71]
/usr/lib64/mysqlrouter/private/libmysqlharness.so.1(+0xdc3b1) [0x7ff3889fb3b1]
/lib64/libpthread.so.0(+0x12d00) [0x7ff388642d00]
/usr/lib64/mysqlrouter/private/libmysqlrouter_routing.so.1(MysqlRoutingClassicConnectionBase::get_server_address[abi:cxx11]() const+0x44) [0x7ff3852e1b14]
/usr/lib64/mysqlrouter/private/libmysqlrouter_routing.so.1(MySQLRouting::get_connections()+0x214) [0x7ff3852b96b4]
/usr/lib64/mysqlrouter/private/libmysqlrouter_routing.so.1(MySQLRoutingAPI::get_connections() const+0x28) [0x7ff3852dc538]
/usr/lib64/mysqlrouter/rest_routing.so(RestRoutingConnections::on_handle_request(HttpRequest&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&)+0x1e0) [0x7ff3853f88d0]
/usr/lib64/mysqlrouter/rest_api.so(RestApi::handle_paths(HttpRequest&)+0x6d2) [0x7ff385481902]
/usr/lib64/mysqlrouter/http_server.so(HttpRequestRouter::route(HttpRequest)+0x15e) [0x7ff38550a9ae]
/usr/lib64/mysqlrouter/http_server.so(+0x39c9b) [0x7ff38550ac9b]
/usr/lib64/mysqlrouter/private/libmysqlrouter_http.so.1(+0xc052) [0x7ff388899052]
/usr/lib64/mysqlrouter/private/libmysqlrouter_http.so.1(+0x1563c) [0x7ff3888a263c]
/usr/lib64/mysqlrouter/private/libmysqlrouter_http.so.1(+0x21f14) [0x7ff3888aef14]
/usr/lib64/mysqlrouter/private/libmysqlrouter_http.so.1(+0x19c3d) [0x7ff3888a6c3d]
/usr/lib64/mysqlrouter/private/libmysqlrouter_http.so.1(+0x19c68) [0x7ff3888a6c68]
/usr/lib64/mysqlrouter/private/libmysqlrouter_http.so.1(+0x2a034) [0x7ff3888b7034]
/usr/lib64/mysqlrouter/private/libmysqlrouter_http.so.1(+0x2a82f) [0x7ff3888b782f]
/usr/lib64/mysqlrouter/http_server.so(HttpRequestThread::wait_and_dispatch()+0x34) [0x7ff3854f7214]
/lib64/libstdc++.so.6(+0xc2ba3) [0x7ff386dc0ba3]
/lib64/libpthread.so.0(+0x81da) [0x7ff3886381da]
/lib64/libc.so.6(clone+0x43) [0x7ff3863d6e73]