Bug #100675 NDB data node disconnted
Submitted: 28 Aug 2020 11:03 Modified: 22 Oct 2020 19:06
Reporter: Phil Brady Email Updates:
Status: Can't repeat Impact on me:
Category:MySQL Cluster: Cluster (NDB) storage engine Severity:S3 (Non-critical)
Version:mysql-5.6.28 ndb-7.4.10 OS:CentOS
Assigned to: MySQL Verification Team CPU Architecture:Any

[28 Aug 2020 11:03] Phil Brady
Last night we had 2 nodes disconnect from our cluster. I have found the error in the logs but do not know what it means.

[09:23] Phil Brady
2020-08-28 05:09:32 [ndbd] INFO     -- /export/home2/pb2/build/sb_0-17731890-1453886244.09/rpm/BUILD/mysql-cluster-gpl-7.4.10/mysql-cluster-gpl-7.4.10/storage/ndb/src/kernel/blocks/dbdih/DbdihMain.cpp
2020-08-28 05:09:32 [ndbd] INFO     -- DBDIH (Line: 19350) 0x00000006
2020-08-28 05:09:32 [ndbd] INFO     -- Error handler shutting down system
2020-08-28 05:09:32 [ndbd] INFO     -- Error handler shutdown completed - exiting
2020-08-28 05:09:33 [ndbd] ALERT    -- Node 8: Forced node shutdown completed. Caused by error 2341: 'Internal program error (failed ndbrequire)(Internal error, programming error or missing error message, please report a bug). Temporary error, restart node'.

I have attached the trace logs. Can some please help me understand the issue?


How to repeat:
Random crash
[28 Aug 2020 11:07] Phil Brady
Sorry I have not attached error report. The file is 123mb and max size is 3mb
[1 Sep 2020 11:54] MySQL Verification Team

Anything "new" that you did recently?

We need whole set of logs, you can please compress and if under 3mb upload to issue or if larger than 3mb you can sftp to our server (look at files tab for details)

[2 Sep 2020 7:20] Phil Brady
Thanks. I have uploaded ndb_error_report_20200828112629.tar.gz to the sftp server.
[22 Oct 2020 19:06] MySQL Verification Team

I cannot figure out exactly why your crash happened but you are running a rather old cluster (latest version of 7.4 is 7.4.30, and you are running 7.4.10), there is a ton of bugfixes already there. You need to upgrade to at least 7.4.30.