Bug #118142 Assertion failure `virtual double ha_ndbcluster::read_time(uint, uint, ha_rows): Assertion 'rows > 0' failed`
Submitted: 8 May 17:00 Modified: 30 May 5:13
Reporter: Congyu Liu (OCA) Email Updates:
Status: Verified Impact on me:
None 
Category:MySQL Cluster: Cluster (NDB) storage engine Severity:S2 (Serious)
Version:8.4.2 OS:Linux
Assigned to: CPU Architecture:x86

[8 May 17:00] Congyu Liu 
Description:
Hi! We found a bug in MySQL NDB API node that triggers the assertion failure `virtual double ha_ndbcluster::read_time(uint, uint, ha_rows): Assertion 'rows > 0' failed`.

How to repeat:
Here's my cluster setup: 1 management node, 2 (replica) data nodes, and 1 API node.

Here's the input (a shell script), which concurrently runs an insert and a transaction of a delete and an update.

```
mysql --socket="/tmp/mysql2.sock" -u "root" --password="" -e "CREATE DATABASE dstestdb; USE dstestdb; CREATE TABLE tnkugncy (kphwizb varchar(1349) NOT NULL,artxual int(19) NULL,iexga int(18) NOT NULL,id int(11) NOT NULL AUTO_INCREMENT,tekjbsu varchar(289) DEFAULT _utf8mb4\"ISSXDELEEPGTCIPJEARPRBUBQYCNY\",tloaa text NULL,cvlegrbb text,fwmohx int(18) NULL,gyhxn varchar(2001) NOT NULL DEFAULT _utf8mb4\"KWZZQRHBHVTFQXWFXEIWJKGVREQKBLVHNKJDGHEWYMPWEWJLTQCFVECHZQGTNGZPAFFXPSCHQOQEGKSWNMHWFMA\",uuid varchar(253),ccdqe text NOT NULL,pjcob datetime DEFAULT '2025-05-31 00:55:40',PRIMARY KEY(id)) ENGINE = NDB DEFAULT CHARACTER SET = utf8mb4 DEFAULT COLLATE = UTF8MB4_BIN; SET GLOBAL debug = 'd:i';"
mysql --socket="/tmp/mysql2.sock" -u "root" --password="" -e "USE dstestdb; INSERT INTO tnkugncy (tnkugncy.fwmohx,tnkugncy.gyhxn,tnkugncy.tloaa,tnkugncy.cvlegrbb,tnkugncy.uuid,tnkugncy.kphwizb,tnkugncy.iexga,tnkugncy.tekjbsu,tnkugncy.ccdqe,tnkugncy.artxual,tnkugncy.pjcob) VALUES (567514099,\"DUKHHBYJQDFULYBVIICDONYDSGOXGQAYBZAIBKOHX\",\"LYUTVIDLOMGQTZNBHTDOXMWTNIEQPEUSDUWBUPSUFXIJAIUIRUROTJMYZWSROHTAWADVDCRCGLULYPJSM\",\"ZORUYAAYRPGRJHJBMOWSTBOOBNKYNKDPRUVHWHQQNDGNSPBYNGOVUSZVJ\",\"67F5BDAE-F74B-45AC-BF97-A6553250892F\",\"DEDZWAJEMCBBAFVDXPIDFNVDYFUHJSHXILYBHSFOHSHBIFKBQVJWKKJPMWTRIIOQVIZPKYANKPTIPSPHZWKJJCAPH\",1988326829,\"AZHSVXOOYAJZSASDBXLFDOGKPVJAOGPOFGNAZXBOWSZZXGXTIPWFQKMBOLSBAPRBZEUE\",\"ACWPJQHTLKGUWPPHQPOXNMNEXKGUTWSMAGADONXYYFFPCVXXPVUMVLLQIP\",5542785,'2088-12-12 18:57:06'); SET GLOBAL debug = 'd:i';" &
mysql --socket="/tmp/mysql2.sock" -u "root" --password="" -e "USE dstestdb; SET TRANSACTION ISOLATION LEVEL READ COMMITTED; BEGIN; DELETE FROM tnkugncy WHERE tnkugncy.artxual<1108105238 XOR tnkugncy.artxual=1469024338; UPDATE tnkugncy SET tnkugncy.iexga=1122588428, tnkugncy.fwmohx=965275993, tnkugncy.kphwizb=\"EDCVLEGRBBRVVNPJCOBCORILFWMOHXJSTORMKPHW\", tnkugncy.cvlegrbb=\"ZBAAJFCARTXUALMDHUNDIEXGALIIFUGYHXNQANWHISSXDELEEPGTCIPJEARPRBUBQYCNYTKWZZQRHB\", tnkugncy.tekjbsu=\"VTFQXWFXEIWJKGVREQKBLVHNKJDGHEWYMPWEWJLTQCFVECHZQGTNGZPAFFXPS\", tnkugncy.tloaa=\"HQOQEGKSWNMHWFMAANAKEEFQJS\", tnkugncy.artxual=84874939 WHERE tnkugncy.tloaa!=_utf8mb4\"ILIWBUPCLGVB\" AND tnkugncy.pjcob='2030-10-27 19:38:15' AND tnkugncy.artxual>2122150880 AND tnkugncy.kphwizb<_utf8mb4\"MVKPRAZHPXEOGAETTLIRUGUWVTGPTJMQXUAACXYPMBBVCAD\" AND tnkugncy.gyhxn=_utf8mb4\"QSYFFBWYYHIDVUWMNHBSNNBNDUGSSOJDFSLRBOGDDZEBDILWLXORLTRCIDPQITHZRUHCBEX\" AND tnkugncy.tekjbsu!=_utf8mb4\"TEHLVDIKXOYECBQDYQQCBLZWUFPOBYDKYVEEMVCQNXKRYJGXDNORDHFETCNNUPYFALNQPZNAOPBXEPQCRBODM\" AND tnkugncy.cvlegrbb!=_utf8mb4\"RPCEPJGFYKVRQZLDYZLPLPKWDQQAOTRRLDYL\" AND tnkugncy.id<59038818; INSERT INTO eefqjs (eefqjs.jmqxuaacx,eefqjs.ydkyv,eefqjs.yiliw,eefqjs.oyecb,eefqjs.idpqith,eefqjs.cadxrqsyf,eefqjs.gvbhkss,eefqjs.blzwu,eefqjs.uuid,eefqjs.dfslrbo,eefqjs.hbsnnbndu,eefqjs.prazhpxe,eefqjs.tlirugu,eefqjs.yyhidv,eefqjs.ebdilwlxo,eefqjs.cbexunteh) VALUES (1629524531,\"EATUQJJHDUKHHBYJQD\",53992151,837148098,\"YBVIICDONYDSGOX\",\"QAYBZAIBKOHXLLYUTVIDLOMGQTZNBHTDOXMWTNIEQPEUSDUWBUPSUFXIJA\",\"\",\"IRUROTJMYZWSROHTAWADVDCRCGLULYPJSMTZORUYAAYRPGRJHJBMOWSTBO\",\"45880CA2-AE3D-46B3-A669-157273FEE233\",'2087-03-02 04:45:24','2037-06-22 00:23:29',1747203263,804148914,'1980-01-08 03:22:55','2078-05-30 13:06:50',484532656),(1941538641,\"RUVHWHQQNDGNSPBYNGOVUSZVJNDEDZWAJEMCBBAFVDXPIDFNVDYFUHJSHXILYBHSFOHSHBIFKBQVJWKKJPMWT\",1704411895,58582792,\"OQVIZPKYANKPTIPSPHZWKJJCAPHVYAZHSVXOOYAJZSASDBXLFDOGKPVJAOGPOFGNAZXBOWSZ\",\"XGXTIPWFQKMBOLSBAPRBZEUEGACWPJQHTLKGUWPPHQPOXNMNEXKGUTWSMAGADONXY\",\"FFPCVXXPVUMVLLQIPBDBWXSWQXHCGNBY\",\"KHPQBHJSR\",\"2DCFA433-37C4-42B0-A849-5C50F7B5E4E4\",'2035-05-15 01:43:31','1977-12-09 15:39:43',254199830,1657239082,'2078-08-16 17:12:26','1974-03-12 05:25:10',213978995),(1966298021,\"VWWRFMRXXPUBEENGOQZRQTXLUEBCZTHCFACLZDXVALYIBNFBLNLMYAEAESSELPIXFJKU\",1463399576,414278294,\"OJGNBDJIKSRPLTJWTTUGHINDGDFTXHMLCZIQBFZ\",\"YFOXXNYFRLOYQJDYVNGQEQMABRJOYPUFAXQJGQQDDQJRKAELAEPISQROWJEBKJTMNJOXFSRFCJVRJOUVLALCNMIYFVTCQCAKHEX\",\"HVZRNDLJVHBQWDLNSWEZBFBZIEEIPOIPPXQLBCAJKRSSRHZCCPGAIARDOFBHAXOQKAAPNYBM\",\"GYJVTIEXJALJHUUSZZTHB\",\"7E574F49-2FE1-44D4-932A-8504266459CF\",'2032-05-08 22:44:49','2073-06-11 06:35:01',906689793,1614163827,'1995-12-02 11:18:15','2061-03-28 23:17:14',929151555),(162215871,\"CUCUQTPPAVKMKJPWMPMIXANTPQKXDDPZPDXYCHIBETSZKVVEZSMJQFOBKSMRSTSWMPIZHITOOIKLCMKTFIJTGHAXHJB\",2006786080,34762921,\"\",\"CZAWIMPILHHORHQESWDCWPBUUXHPCAAZJTJIKWNUEJUUGRIOK\",\"RDUYTWGOTWXAVQVKLHTISWFYHKKWLNDIQUCLGULEAGZTLUGBPCIGPJTOQRWHV\",\"JUVWZNPIWISUDDOWPVAZMVDWPDSQADEDVCKEOQ\",\"5BEF1814-FEEA-4C6A-9DDD-0D4BF0DB52E8\",'2011-03-23 03:56:03','2029-03-15 11:46:28',235504638,147053564,'2074-09-09 17:25:16','2061-09-19 17:38:26',1339805847),(1779712310,\"USNCIBDXGAROZBSNNNRCUMDHMTWLOBULYKSBZNKVXJMUFDVDULBVKNGQYHVXMPNFZWJYMTNWIRML\",549387978,1244551629,\"THS\",\"PEPAVVGANIPN\",\"FEXQXTQMTKUMZFQUSCECKFVGHYYDQMNAKMBFRHDSIQRGTCHCIRMYL\",\"JBEUBYVBYEYMBBUDPMZPMFHWASFFWZGOTFJ\",\"CF40BD1A-7BF7-42C8-83E9-844DB1AFE289\",'1977-03-21 00:11:08','2063-06-26 04:10:00',367109981,2065911118,'2033-12-16 01:09:18','2033-06-08 23:56:56',1257814650),(1087246805,\"QLWGEQYCEWRVWEPWTNKRVRFIBGJZYTQLUIRRNOALMVBSKZKZGADLVACHXYSKOJ\",2117592037,1500285318,\"JOVKWZHKFAUFTJVDTJTQUF\",\"PRGKEWQXZKUNGNJOXMRRMWCCAADIOHLQUOVWQTGHLHWVJZBCHHAZAVAWPIZGCRXZASWHTOJJNJM\",\"JSKYDNBVZFFUUJBVUGHIDGJDIHUKIECPMGKMTQDDJIHKFNQHN\",\"UTIGYMWBYXJVYUUERUFLJYNKXDUMXDMPLIAQHSHOVGGTKUEESTPCLFGVJUKTH\",\"864C5080-63A5-4BA2-99AB-45CB5877D156\",'2082-12-29 23:11:40','2014-08-07 16:08:50',1473459441,336317552,'2069-03-29 13:28:35','1971-08-26 10:39:55',148134553),(1912267577,\"WXQL\",1518608424,2132726702,\"HESVPFVNAHMEIDZFSKLWRGJVBOBWUMNUMESIAOHKQVEPAENMAEGHDHRETIBANNGBPKBOUNHN\",\"IYFKUWMKEPLS\",\"RRVHIMAEZOYLMXOINP\",\"CWQITSYOKCRJX\",\"75BAE666-CEE6-4D95-9950-E4C51461E0AE\",'1970-03-23 00:07:36','2090-08-04 11:00:46',2072359262,373472538,'2077-01-04 21:39:50','2048-04-25 05:11:19',245440769),(758638607,\"JFFITPNCVRWGDALHVQFXLJLTRWHXQXVVZLSTQWDLJGIFTXG\",1633731573,1862316629,\"IMOHLHFQIJFQKTJGFXLIFEQOFDUFONKBPSKUIOBXFVPVWSGQQGCGUAWANBQCLTAIALFRWIRYIRI\",\"BBAMHSPFCFYIUGPDJTNJKKPWOBYNFIA\",\"KZDUELCJWOFWWVITVTCPTQAWSOYCHRKFNBWRSCJNAZPDAZAVAYX\",\"JQJPDWGOKAKAVOOZAHDYQLMEXRHUDINMBQTWPDAFNUKJUIQILJJEQAYRIUSTIIVBZKGIIWOZNNFDL\",\"951D5B00-D222-45F2-89D5-1B6D1BFF65AF\",'1987-12-17 23:37:05','2075-07-10 04:13:31',218926244,1516007873,'2021-04-06 21:13:54','2006-02-09 18:11:21',1134613801); ROLLBACK; COMMIT;" &
wait
```

Callstack on crash:
```
 #0 0x55555907283b print_fatal_signal at /mysql-server/sql/signal_handler.cc:319
 #1 0x555559072aca _Z19handle_fatal_signaliP9siginfo_tPv at /mysql-server/sql/signal_handler.cc:399
 #2 0x7ffff744251f <unknown> at sysdeps/unix/sysv/linux/x86_64/libc_sigaction.c:0
 #3 0x7ffff74969fc __pthread_kill_implementation at ./nptl/pthread_kill.c:44
 #4 0x7ffff74969fc __pthread_kill_internal at ./nptl/pthread_kill.c:78
 #5 0x7ffff74969fc __GI___pthread_kill at ./nptl/pthread_kill.c:89
 #6 0x7ffff7442475 __GI_raise at sysdeps/posix/raise.c:26
 #7 0x7ffff74287f2 __GI_abort at ./stdlib/abort.c:79
 #8 0x7ffff742871a __assert_fail_base at ./assert/assert.c:94
 #9 0x7ffff7439e95 __GI___assert_fail at ./assert/assert.c:103
 #10 0x55555b054875 _ZN13ha_ndbcluster9read_timeEjjy at /mysql-server/storage/ndb/plugin/ha_ndbcluster.cc:7212
 #11 0x555559251d0f _ZN7handler9read_costEjdd at /mysql-server/sql/handler.cc:6152
 #12 0x5555592526fe _ZN7handler27multi_range_read_info_constEjP12RANGE_SEQ_IFPvjPjS3_P13Cost_estimate at /mysql-server/sql/handler.cc:6363
 #13 0x55555b068c45 _ZN13ha_ndbcluster27multi_range_read_info_constEjP12RANGE_SEQ_IFPvjPjS3_P13Cost_estimate at /mysql-server/storage/ndb/plugin/ha_ndbcluster.cc:13310
 #14 0x555559591ff0 _Z18check_quick_selectP3THDP15RANGE_OPT_PARAMjbP8SEL_ROOTb10enum_orderbPjS6_P13Cost_estimatePbS9_ at /mysql-server/sql/range_optimizer/index_range_scan_plan.cc:627
 #15 0x555559592da1 _Z20get_key_scans_paramsP3THDP15RANGE_OPT_PARAMP8SEL_TREEbb10enum_orderbdbP6BitmapILj64EE at /mysql-server/sql/range_optimizer/index_range_scan_plan.cc:868
 #16 0x555558c374e0 _Z17test_quick_selectP3THDP8MEM_ROOTS2_6BitmapILj64EEmmyb10enum_orderP5TABLEbP4ItemPS4_bP11Query_blockPP10AccessPath at /mysql-server/sql/range_optimizer/range_optimizer.cc:688
 #17 0x555558f56a9d _ZN14Sql_cmd_update19update_single_tableEP3THD at /mysql-server/sql/sql_update.cc:525
 #18 0x555558f5b5e7 _ZN14Sql_cmd_update13execute_innerEP3THD at /mysql-server/sql/sql_update.cc:1823
 #19 0x555558e8fee3 _ZN11Sql_cmd_dml7executeEP3THD at /mysql-server/sql/sql_select.cc:782
 #20 0x555558dfc93c _Z21mysql_execute_commandP3THDb at /mysql-server/sql/sql_parse.cc:3670
 #21 0x555558e026ff _Z20dispatch_sql_commandP3THDP12Parser_state at /mysql-server/sql/sql_parse.cc:5392
 #22 0x555558df7f3d _Z16dispatch_commandP3THDPK8COM_DATA19enum_server_command at /mysql-server/sql/sql_parse.cc:2136
 #23 0x555558df5c9e _Z10do_commandP3THD at /mysql-server/sql/sql_parse.cc:1465
 #24 0x55555905b838 handle_connection at /mysql-server/sql/conn_handler/connection_handler_per_thread.cc:304
 #25 0x55555b422764 pfs_spawn_thread at /mysql-server/storage/perfschema/pfs.cc:3061
 #26 0x7ffff7494ac2 start_thread at ./nptl/pthread_create.c:442
 #27 0x7ffff752684f <unknown> at sysdeps/unix/sysv/linux/x86_64/clone3.S:81
 #28 0xffffffffffffffff <unknown>
```

Suggested fix:
The bug is potentially related to `stats.records`. I noticed in the log that:
```
# from ha_ndbcluster::info(uint)
T@13: int ha_ndbcluster::info: table_stats: records updated from trans stats: 18446744073709551615 # -1
```

However, in `ha_ndbcluster::info` function, it seems that it tries to ensure `stats.records` is never less than 2:
```
      // Adjust `stats.records` to never be < 2 since optimizer interprets the
      // values 0 and 1 as EXACT.
      // NOTE! It looks like the above statement is correct only when
      // HA_STATS_RECORDS_IS_EXACT is returned from table_flags(), something
      // which ndbcluster does not.
      if (stats.records < 2) {
        DBUG_PRINT("table_stats", ("adjust records %llu -> 2", stats.records));
        stats.records = 2;
      }
```

But since -1 is interpreted as a uint, it never made it into the branch. This `stats.records` later affects how the `total_rows` is calculated in `multi_range_read_info_const`.
[12 May 19:01] MySQL Verification Team 
Thank you very much for your patch contribution, we appreciate it!

In order for us to continue the process of reviewing your contribution to MySQL, please send us a signed copy of the Oracle Contributor Agreement (OCA) as outlined in https://oca.opensource.oracle.com

Signing an OCA needs to be done only once and it's valid for all other Oracle governed Open Source projects as well.

Getting a signed/approved OCA on file will help us facilitate your contribution - this one, and others in the future.  

Please let me know, if you have any questions.

Thank you for your interest in MySQL.
[12 May 20:00] Congyu Liu 
Hi. I just signed the OCA. Let me know if there's anything else that I need to do.
[12 May 20:12] MySQL Verification Team 
Hi,

Uploading the result of ndb_error_reporter would be very useful to us.

Thanks
[12 May 20:48] Congyu Liu 
Uploaded. It seems that the tarball only contains the mgmd config. I did see errors like `scp .../ndbd2/ndb_3_fs: not a regular file` when running the ndb_error_report, not sure if that could be a problem.
[12 May 21:19] MySQL Verification Team 
Those scp's were supposed to collect all error logs from all nodes.

If ndb_error_reporter did not collect them, maybe you can collect all error logs and upload them please.

Thanks
[30 May 5:13] Congyu Liu 
Hello,

Sorry for the late reply. I just uploaded the log files. Hope they help.