Bug #84651 unable to start ndbmtd after upgrade
Submitted: 24 Jan 2017 20:07 Modified: 31 Jan 2017 5:15
Reporter: Jay Jay Email Updates:
Status: Verified Impact on me:
Category:MySQL Cluster: Cluster (NDB) storage engine Severity:S3 (Non-critical)
Version:7.5.5 OS:Linux (x64)
Assigned to: CPU Architecture:Any
Tags: ndbmtd

[24 Jan 2017 20:07] Jay Jay
2 node 2 copy setup. Small database.
Node 1: Data usage is 13%(340 32K pages of total 2560)
Node 1: Index usage is 5%(122 8K pages of total 2336)

System running 7.4.12 trying to upgrade to 7.5.5

ndb_mgmd restarted ok with 7.5.5
mysqld ok with 7.5.5
ndbmtd however shortly just exits with reason

Time: Monday 23 January 2017 - 22:37:23
Status: Temporary error, restart node
Message: WatchDog terminate, internal error or massive overload on the machine running this node (Internal error, programming error or missing error message, please report a bug)
Error: 6050
Error data: Allocating memory
Error object: /jlinux/source/mysql/7.5.5/mysql-cluster-gpl-7.5.5/storage/ndb/src/kernel/vm/WatchDog.cpp
Program: ndbmtd
Pid: 3213
Version: mysql-5.7.17 ndb-7.5.5
Trace file name: ndb_2_trace.log.4
Trace file path: /var/app/mysql/ndb/ndb_2_trace.log.4 [t1..t4]

If I change ngbmgmd config to
TimeBetweenWatchDogCheckInitial = 120000
TimeBetweenWatchDogCheck = 120000

the same thing happens, but it takes a longer time.
ndbmtd process is also using 100% cpu

How to repeat:
upgrade ndb from 7.4.12 to 7.5.5

ndb config is empty / default mysql values
[24 Jan 2017 20:08] Jay Jay

Attachment: ndb_2_trace.log.4 (application/octet-stream, text), 22.01 KiB.

[24 Jan 2017 20:11] Jay Jay
The allocating memory error message is not due to not enough ram. Even if I increase from the original 2gb ram to 64gb ram the same error happens.
[31 Jan 2017 5:15] MySQL Verification Team

Thanks for the report, I did manage to verify this bug as described.

best regards