Bug #94180 NDBD crash on startup whilst rebuilding ordered indexes
Submitted: 2 Feb 2019 19:36 Modified: 5 Feb 2019 1:08
Reporter: Joshua Gigg Email Updates:
Status: Not a Bug Impact on me:
None 
Category:MySQL Cluster: Cluster (NDB) storage engine Severity:S2 (Serious)
Version:7.6.9 OS:CentOS (7)
Assigned to: MySQL Verification Team CPU Architecture:x86 (x86-64)
Tags: ndb

[2 Feb 2019 19:36] Joshua Gigg 
Description:
We've got a very small cluster (with maybe 100MB of data in it) with 4 data nodes (in two node groups).

All was fine originally, but now if we try to restart a data node, we get an error when attempting to rebuild ordered indexes.

2019-02-02 19:00:38 [ndbd] INFO     -- LDM(6): Starting to rebuild ordered indexes
2019-02-02 19:00:38 [ndbd] INFO     -- LDM(4): Completed old Redo head invalidation on log part 3
2019-02-02 19:00:38 [ndbd] INFO     -- LDM(4): All redo actions complete (apply, invalidate)
2019-02-02 19:00:38 [ndbd] INFO     -- LDM(4): Starting to rebuild ordered indexes
2019-02-02 19:00:38 [ndbd] INFO     -- LDM(18): index id 37 rebuild done
2019-02-02 19:00:38 [ndbd] INFO     -- LDM(11): index id 9 rebuild done
2019-02-02 19:00:38 [ndbd] INFO     -- LDM(15): index id 30 rebuild done
2019-02-02 19:00:38 [ndbd] INFO     -- LDM(17): index id 42 rebuild done
2019-02-02 19:00:39 [ndbd] INFO     -- /export/home2/pb2/build/sb_1-32108591-1546542676.1/rpm/BUILD/mysql-cluster-gpl-7.6.9/mysql-cluster-gpl-7.6.9/storage/ndb/src/kernel/blocks/ndbfs/Ndbfs.cpp
2019-02-02 19:00:39 [ndbd] INFO     -- NDBFS (Line: 1181) 0x00000002 Check false failed
2019-02-02 19:00:39 [ndbd] INFO     -- Error handler shutting down system
2019-02-02 19:00:39 [ndbd] INFO     -- Error handler shutdown completed - exiting
2019-02-02 19:00:40 [ndbd] ALERT    -- Node 4: Forced node shutdown completed. Occured during startphase 5. Caused by error 2341: 'Internal program error (failed ndbrequire)(Internal error, programming error or missing error message, please report a bug). Temporary error, restart node'.

We've tried this with 7.6.8 and 7.6.9. We also have this issue with our second cluster that ndb_restored the data from the first.

How to repeat:
N/A
[2 Feb 2019 19:41] Joshua Gigg 
GDB Backtrace from 7.6.8 (same issue)

Attachment: gdb-output.txt (text/plain), 11.78 KiB.
[3 Feb 2019 10:27] Joshua Gigg 
Have tried with an empty cluster (--initial) on all nodes, running the distributed privileges script, then attempting to restart one node. Same issue.
[4 Feb 2019 9:22] Joshua Gigg 
Before distributed privileges: startup fine
After distributed privileges: crash

Following standard instructions from: https://dev.mysql.com/doc/mysql-cluster-excerpt/5.5/en/mysql-cluster-privilege-distributio...
[4 Feb 2019 16:04] Joshua Gigg 
Our SharedGlobalMemory was set to 20M. Increasing this to 4G fixed the problem.

Left over config from an older MySQL Cluster version.
[5 Feb 2019 1:08] MySQL Verification Team 
Hi,

What can I say, you asked the question and answered yourself :D

all best
Bogdan