| Bug #114850 | lost connection in MySQL NDB cluster | ||
|---|---|---|---|
| Submitted: | 2 May 2024 10:13 | Modified: | 8 May 2024 6:25 |
| Reporter: | CunDi Fang | Email Updates: | |
| Status: | Duplicate | Impact on me: | |
| Category: | MySQL Cluster: Cluster (NDB) storage engine | Severity: | S2 (Serious) |
| Version: | 8.0.35-cluster MySQL Cluster Community S | OS: | Any |
| Assigned to: | MySQL Verification Team | CPU Architecture: | Any |
[7 May 2024 8:24]
MySQL Verification Team
You need to upload clean ndb_error_reporter result. Clean the logs before you restart cluster and reproduce the problem, upload logs. Thanks
[8 May 2024 5:16]
MySQL Verification Team
Hi, There is no crash in these logs. Can you upload mysql error log from the mysql server you were connected to. Thanks
[8 May 2024 5:53]
CunDi Fang
I seem to know what you want, do you need these log messages below? ``` 2024-05-08T05:50:32.773822Z 0 [System] [MY-010866] [NDB] Metadata: Failed to submit table 'mysql.ndb_apply_status' for synchronization mbind: Operation not permitted 2024-05-08T05:50:53Z UTC - mysqld got signal 11 ; Most likely, you have hit a bug, but this error can also be caused by malfunctioning hardware. BuildID[sha1]=1135254ad6e65d8565c7a1bb98c0a03b430d0f46 Thread pointer: 0x7f0c7c000b70 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 7f0cfc575b70 thread_stack 0x100000 /usr/local/mysql-cluster/bin/mysqld(my_print_stacktrace(unsigned char const*, unsigned long)+0x41) [0x2134dc1] /usr/local/mysql-cluster/bin/mysqld(print_fatal_signal(int)+0x2a2) [0xff07a2] /usr/local/mysql-cluster/bin/mysqld(handle_fatal_signal+0xa5) [0xff0955] /lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7f0d2e5ee520] /usr/local/mysql-cluster/bin/mysqld(ndb_pushed_builder_ctx::make_pushed_join()+0xb1) [0x2640c11] /usr/local/mysql-cluster/bin/mysqld(ndbcluster_push_to_engine(THD*, AccessPath*, JOIN*)+0x264) [0x2623fb4] /usr/local/mysql-cluster/bin/mysqld(JOIN::push_to_engines()+0x71) [0xe5ec01] /usr/local/mysql-cluster/bin/mysqld(JOIN::optimize(bool)+0x1cf8) [0xe78978] /usr/local/mysql-cluster/bin/mysqld(Query_block::optimize(THD*, bool)+0xc1) [0xedac91] /usr/local/mysql-cluster/bin/mysqld(Query_expression::optimize(THD*, TABLE*, bool, bool)+0xad) [0xf5990d] /usr/local/mysql-cluster/bin/mysqld(Sql_cmd_dml::execute_inner(THD*)+0x34) [0xed9bb4] /usr/local/mysql-cluster/bin/mysqld(Sql_cmd_dml::execute(THD*)+0x1e4) [0xee4ef4] /usr/local/mysql-cluster/bin/mysqld(mysql_execute_command(THD*, bool)+0xae7) [0xe80bc7] /usr/local/mysql-cluster/bin/mysqld(dispatch_sql_command(THD*, Parser_state*)+0x51b) [0xe843bb] /usr/local/mysql-cluster/bin/mysqld(dispatch_command(THD*, COM_DATA const*, enum_server_command)+0x2351) [0xe86d01] /usr/local/mysql-cluster/bin/mysqld(do_command(THD*)+0x15b) [0xe8787b] /usr/local/mysql-cluster/bin/mysqld() [0xfe09b8] /usr/local/mysql-cluster/bin/mysqld() [0x2848944] /lib/x86_64-linux-gnu/libc.so.6(+0x94ac3) [0x7f0d2e640ac3] /lib/x86_64-linux-gnu/libc.so.6(clone+0x44) [0x7f0d2e6d1bf4] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (7f0c7c006510): is an invalid pointer Connection ID (thread ID): 21 Status: NOT_KILLED The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. 2024-05-08T05:50:53.474384Z 0 [Warning] [MY-011070] [Server] 'Disabling symbolic links using --skip-symbolic-links (or equivalent) is the default. Consider not using this option as it' is deprecated and will be removed in a future release. 2024-05-08T05:50:53.474528Z 0 [System] [MY-010116] [Server] /usr/local/mysql-cluster/bin/mysqld (mysqld 8.0.35-cluster) starting as process 10404 2024-05-08T05:50:53.483788Z 1 [System] [MY-013576] [InnoDB] InnoDB initialization has started. 2024-05-08T05:50:53.783298Z 1 [System] [MY-013577] [InnoDB] InnoDB initialization has ended. 2024-05-08T05:50:53.877668Z 0 [ERROR] [MY-010901] [Server] Can't open shared library '/usr/local/mysql-cluster/lib/plugin/' (errno: 22 /usr/local/mysql-cluster/lib/plugin/: cannot read file data: Is a directory). 2024-05-08T05:50:53.877714Z 0 [Warning] [MY-010736] [Server] Couldn't load plugin named '' with soname ''. 2024-05-08T05:50:53.879844Z 0 [System] [MY-010866] [NDB] Metadata: Initialization 2024-05-08T05:50:57.696106Z 0 [System] [MY-010865] [NDB] connection[0], NodeID: 8, all storage nodes connected 2024-05-08T05:50:57.696452Z 0 [System] [MY-010866] [NDB] Binlog: Starting... 2024-05-08T05:50:57.696609Z 2 [System] [MY-010866] [NDB] Binlog: Started 2024-05-08T05:50:57.696624Z 2 [System] [MY-010866] [NDB] Binlog: Setting up 2024-05-08T05:50:57.696771Z 0 [System] [MY-010866] [NDB] Index Stat: Starting... 2024-05-08T05:50:57.696823Z 0 [System] [MY-010866] [NDB] Index Stat: Wait for server start 2024-05-08T05:50:57.696838Z 2 [System] [MY-010866] [NDB] Binlog: Setup completed 2024-05-08T05:50:57.696852Z 2 [System] [MY-010866] [NDB] Binlog: Wait for server start 2024-05-08T05:50:57.696947Z 0 [System] [MY-010866] [NDB] Metadata: Starting... 2024-05-08T05:50:57.696969Z 0 [System] [MY-010866] [NDB] Metadata: Wait for server start 2024-05-08T05:50:57.701881Z 0 [System] [MY-010865] [NDB] ndbinfo prefix: './ndbinfo/ndb@0024' 2024-05-08T05:50:57.787168Z 0 [System] [MY-010229] [Server] Starting XA crash recovery... 2024-05-08T05:50:57.799758Z 0 [System] [MY-010232] [Server] XA crash recovery finished. 2024-05-08T05:50:57.897018Z 0 [Warning] [MY-010068] [Server] CA certificate ca.pem is self signed. 2024-05-08T05:50:57.897062Z 0 [System] [MY-013602] [Server] Channel mysql_main configured to support TLS. Encrypted connections are now supported for this channel. ``` These error.log files from SQL nodes don't seem to be available via "ndb_error_reporter".
[8 May 2024 6:25]
MySQL Verification Team
Hi, Yes, thanks, this is duplicate of Bug #114777
Description: Hello, again a lost connection error in NDB cluster Poc: ``` select ref_0.column3 as c0, subq_0.c0 as c1, subq_0.c0 as c2, subq_0.c0 as c3 from mytest90.test2 as ref_0 right join (select ref_1.column1 as c0, ref_1.column1 as c1 from mytest90.test6 as ref_1 where EXISTS ( select ref_2.column7 as c0, ref_2.column5 as c1, ref_2.column5 as c2, ref_1.column1 as c3, ref_2.column9 as c4, ref_1.column1 as c5, (select io_latency from sys.x$host_summary_by_file_io limit 1 offset 5) as c6, ref_2.column5 as c7, ref_2.column2 as c8, ref_2.column1 as c9, ref_1.column1 as c10, ref_1.column1 as c11 from mytest90.test0 as ref_2 where ref_1.column1 is not NULL limit 177) limit 95) as subq_0 on (false), lateral (select subq_0.c0 as c0, ref_0.column2 as c1, ref_0.column66 as c2 from (select ref_0.column3 as c0, ref_0.column1 as c1, ref_4.column6 as c2, 39 as c3, (select THREAD_ID from performance_schema.events_waits_summary_by_thread_by_event_name limit 1 offset 4) as c4, ref_4.column4 as c5 from mytest90.test6 as ref_3 left join mytest90.test7 as ref_4 on (false) where false limit 123) as subq_1 right join mytest90.test2 as ref_5 on (false) where (false) or (((false) or (false)) or (true)) limit 116) as subq_2 where (true) or (EXISTS ( select ref_0.column1 as c0, subq_2.c1 as c1, ref_0.column2 as c2, ref_0.column3 as c3, ref_0.column66 as c4 from mytest90.test9 as ref_6 where (subq_0.c1 is not NULL) and ((ref_6.column5 is NULL) or (subq_0.c0 is not NULL)))) limit 115; ``` Architecture Information: ''' [NDBD DEFAULT] NoOfReplicas =2 DataMemory = 512M IndexMemory = 64M [NDB_MGMD] NodeId=1 hostname =192.172.10.8 datadir =/var/lib/mysql-cluster [NDBD] NodeId =2 hostname =192.172.10.9 datadir =/usr/local/mysql-cluster/data NodeGroup=0 [NDBD] NodeId =3 hostname =192.172.10.10 datadir =/usr/local/mysql-cluster/data NodeGroup=1 [NDBD] NodeId =4 hostname =192.172.10.11 datadir =/usr/local/mysql-cluster/data NodeGroup=0 [NDBD] NodeId =5 hostname =192.172.10.12 datadir =/usr/local/mysql-cluster/data NodeGroup=1 [mysqld] NodeId =6 hostname =192.172.10.9 [mysqld] NodeId =7 hostname =192.172.10.10 [mysqld] NodeId =8 hostname =192.172.10.11 [mysqld] NodeId =9 hostname =192.172.10.12 ''' Attempted and successfully reproduced! If it is same with BUG#114464? How to repeat: Simply execute that PoC to trigger it. Suggested fix: If possible, can you tell me what is the root cause of this bug? You only said before that this select is not suitable for execution on NDB, can you tell me why? So that I can avoid similar sql statements.