Bug #81763 ndb_desc output should be deterministic independent of platform
Submitted: 7 Jun 2016 21:31 Modified: 27 Jun 2016 11:40
Reporter: Mauritz Sundell Email Updates:
Status: Closed Impact on me:
None 
Category:Tests: Cluster Severity:S3 (Non-critical)
Version: OS:Any
Assigned to: CPU Architecture:Any

[7 Jun 2016 21:31] Mauritz Sundell
Description:
ndb_desc are used in several MTR test for ndb.

ndb_desc output is not deterministic, indexes can be listed in different order on different platforms.
This becomes a problem for test there ndb_desc output is recorded.

An other related problem is that table and index id might differ between runs of the test, and since the id is visible in different places in ndb_desc output that too is a source of indeterminism and a problem for recorded tests.

Some of the indeterminism have been coped with in MTR using replace_regex, but that does not help for the current failures of ndb.ndb_fully_replicated_restart.

How to repeat:
On OS X 10 index occasionally are listed in different order than recorded on Linux.

ndb.ndb_fully_replicated_restart         w2 [ fail ]
        Test ended at 2016-06-04 11:52:32

CURRENT_TEST: ndb.ndb_fully_replicated_restart
--- /hd3/pb2/test/sb_2-19177065-1465033011.14/mysql-cluster-gpl-7.5.3-osx10.11-x86_64/mysql-test/suite/ndb/r/ndb_fully_replicated_restart.result	2016-06-03 19:26:58.000000000 +0300
+++ /Volumes/ramdisk/mtr-29252/var-n_mix/2/log/ndb_fully_replicated_restart.reject	2016-06-04 12:52:31.000000000 +0300
@@ -42,10 +42,10 @@
 c Int NULL AT=FIXED ST=MEMORY DYNAMIC
 -- Indexes -- 
 PRIMARY KEY(a, b) - UniqueHashIndex
+c$unique(c) - UniqueHashIndex
 b(b) - OrderedIndex
 c(c) - OrderedIndex
 PRIMARY(a, b) - OrderedIndex
-c$unique(c) - UniqueHashIndex
 
 NDBT_ProgramExit: 0 - OK

In mysql-test/suite/ndb/include/ndb_desc_print.inc there is some hacks making ndb_desc output more deterministic.

# Use replace_regex to remove non-deterministic values from the
# ndb_desc output
#
--replace_regex /Version: [0-9]*/Version: Any/  /t2_unique_index\(j\) - OrderedIndex/Index/ /t2_unique_index\$uni
que\(j\) - UniqueHashIndex/Index/ /PRIMARY\(i\) - OrderedIndex/Index/  /NDB\$BLOB_[0-9]*_/NDB$BLOB_XX_/  /Length
of frm data: [0-9]*/Length of frm data: XXX/
--exec $ndb_desc_cmd

Suggested fix:
Add options to ndb_desc making output more deterministic and use them in MTR tests.
1) allow sorting indexes for a table by name
2) replace visible id like for blob table names (NDB$BLOB_id_colname) in column definition with either some constant string like # or object name, that is NDB$BLOB_#_colname or NDB$BLOB_{db/def/tbl}

An alternative would be to not use ndb_desc but some other tool or ndbinfo if they provide enough information.
[14 Jun 2016 14:51] Mauritz Sundell
Posted by developer:
 
ndb_fully_replicated_restart fails also on Linux if it runs after ndb_index_stat_restart.

Reason is that ndb_index_stat_restarts creates an extra hash map due to creating a table with one partition.
Indexes created in ndb_fully_replicated_restart will get an id one higher than if the extra hashmap was not created, and the internal key in hash list of indexes will get another order.

By listing index in id order the index order will be stable.

Note, the extra hashmap is not visible from sql so mysqltest post test checks do not discover this change.  And also there are currently no way to drop a hashmap.
[27 Jun 2016 11:40] Jon Stephens
Documented fix in the NDB 7.5.3 changelog as follows:

    Table indexes were listed in the output of ndb_desc in a
    undefined order that could vary between platforms. Now these
    indexes are ordered by ID.

Closed.