Bug #100675 NDB data node disconnted
Submitted: 28 Aug 11:03 Modified: 22 Oct 19:06
Reporter: Phil Brady Email Updates:
Status: Can't repeat Impact on me:
None 
Category:MySQL Cluster: Cluster (NDB) storage engine Severity:S3 (Non-critical)
Version:mysql-5.6.28 ndb-7.4.10 OS:CentOS
Assigned to: MySQL Verification Team CPU Architecture:Any

[28 Aug 11:03] Phil Brady
Description:
Last night we had 2 nodes disconnect from our cluster. I have found the error in the logs but do not know what it means.

[09:23] Phil Brady
    
2020-08-28 05:09:32 [ndbd] INFO     -- /export/home2/pb2/build/sb_0-17731890-1453886244.09/rpm/BUILD/mysql-cluster-gpl-7.4.10/mysql-cluster-gpl-7.4.10/storage/ndb/src/kernel/blocks/dbdih/DbdihMain.cpp
2020-08-28 05:09:32 [ndbd] INFO     -- DBDIH (Line: 19350) 0x00000006
2020-08-28 05:09:32 [ndbd] INFO     -- Error handler shutting down system
2020-08-28 05:09:32 [ndbd] INFO     -- Error handler shutdown completed - exiting
2020-08-28 05:09:33 [ndbd] ALERT    -- Node 8: Forced node shutdown completed. Caused by error 2341: 'Internal program error (failed ndbrequire)(Internal error, programming error or missing error message, please report a bug). Temporary error, restart node'.

I have attached the trace logs. Can some please help me understand the issue?

Thanks

How to repeat:
Random crash
[28 Aug 11:07] Phil Brady
Sorry I have not attached error report. The file is 123mb and max size is 3mb
[1 Sep 11:54] MySQL Verification Team
Hi,

Anything "new" that you did recently?

We need whole set of logs, you can please compress and if under 3mb upload to issue or if larger than 3mb you can sftp to our server (look at files tab for details)

Thanks
Bogdan
[2 Sep 7:20] Phil Brady
Thanks. I have uploaded ndb_error_report_20200828112629.tar.gz to the sftp server.
[22 Oct 19:06] MySQL Verification Team
Hi,

I cannot figure out exactly why your crash happened but you are running a rather old cluster (latest version of 7.4 is 7.4.30, and you are running 7.4.10), there is a ton of bugfixes already there. You need to upgrade to at least 7.4.30.

Thanks
Bogdan