Bug #94180 NDBD crash on startup whilst rebuilding ordered indexes

Description:
We've got a very small cluster (with maybe 100MB of data in it) with 4 data nodes (in two node groups).

All was fine originally, but now if we try to restart a data node, we get an error when attempting to rebuild ordered indexes.

2019-02-02 19:00:38 [ndbd] INFO     -- LDM(6): Starting to rebuild ordered indexes
2019-02-02 19:00:38 [ndbd] INFO     -- LDM(4): Completed old Redo head invalidation on log part 3
2019-02-02 19:00:38 [ndbd] INFO     -- LDM(4): All redo actions complete (apply, invalidate)
2019-02-02 19:00:38 [ndbd] INFO     -- LDM(4): Starting to rebuild ordered indexes
2019-02-02 19:00:38 [ndbd] INFO     -- LDM(18): index id 37 rebuild done
2019-02-02 19:00:38 [ndbd] INFO     -- LDM(11): index id 9 rebuild done
2019-02-02 19:00:38 [ndbd] INFO     -- LDM(15): index id 30 rebuild done
2019-02-02 19:00:38 [ndbd] INFO     -- LDM(17): index id 42 rebuild done
2019-02-02 19:00:39 [ndbd] INFO     -- /export/home2/pb2/build/sb_1-32108591-1546542676.1/rpm/BUILD/mysql-cluster-gpl-7.6.9/mysql-cluster-gpl-7.6.9/storage/ndb/src/kernel/blocks/ndbfs/Ndbfs.cpp
2019-02-02 19:00:39 [ndbd] INFO     -- NDBFS (Line: 1181) 0x00000002 Check false failed
2019-02-02 19:00:39 [ndbd] INFO     -- Error handler shutting down system
2019-02-02 19:00:39 [ndbd] INFO     -- Error handler shutdown completed - exiting
2019-02-02 19:00:40 [ndbd] ALERT    -- Node 4: Forced node shutdown completed. Occured during startphase 5. Caused by error 2341: 'Internal program error (failed ndbrequire)(Internal error, programming error or missing error message, please report a bug). Temporary error, restart node'.

We've tried this with 7.6.8 and 7.6.9. We also have this issue with our second cluster that ndb_restored the data from the first.

How to repeat:
N/A

GDB Backtrace from 7.6.8 (same issue)

Attachment: gdb-output.txt (text/plain), 11.78 KiB.

Have tried with an empty cluster (--initial) on all nodes, running the distributed privileges script, then attempting to restart one node. Same issue.

Before distributed privileges: startup fine
After distributed privileges: crash

Following standard instructions from: https://dev.mysql.com/doc/mysql-cluster-excerpt/5.5/en/mysql-cluster-privilege-distributio...

Our SharedGlobalMemory was set to 20M. Increasing this to 4G fixed the problem.

Left over config from an older MySQL Cluster version.

Hi,

What can I say, you asked the question and answered yourself :D

all best
Bogdan