Bug #84651 unable to start ndbmtd after upgrade

• Submitted: 24 Jan 2017 20:07
• Modified: 31 Jan 2017 5:15
• Reporter: Jay Jay
• Status: Verified
• Category: MySQL Cluster: Cluster (NDB) storage engine
• Severity: S3 (Non-critical)
• Version: 7.5.5
• OS: Linux (x64)
• CPU Architecture: Any
• Tags: ndbmtd

[24 Jan 2017 20:07] Jay Jay

Description:
2 node 2 copy setup. Small database.
Node 1: Data usage is 13%(340 32K pages of total 2560)
Node 1: Index usage is 5%(122 8K pages of total 2336)

System running 7.4.12 trying to upgrade to 7.5.5

ndb_mgmd restarted ok with 7.5.5
mysqld ok with 7.5.5
ndbmtd however shortly just exits with reason

Time: Monday 23 January 2017 - 22:37:23
Status: Temporary error, restart node
Message: WatchDog terminate, internal error or massive overload on the machine running this node (Internal error, programming error or missing error message, please report a bug)
Error: 6050
Error data: Allocating memory
Error object: /jlinux/source/mysql/7.5.5/mysql-cluster-gpl-7.5.5/storage/ndb/src/kernel/vm/WatchDog.cpp
Program: ndbmtd
Pid: 3213
Version: mysql-5.7.17 ndb-7.5.5
Trace file name: ndb_2_trace.log.4
Trace file path: /var/app/mysql/ndb/ndb_2_trace.log.4 [t1..t4]
***EOM***

If I change ngbmgmd config to
TimeBetweenWatchDogCheckInitial = 120000
TimeBetweenWatchDogCheck = 120000

the same thing happens, but it takes a longer time.
ndbmtd process is also using 100% cpu

How to repeat:
upgrade ndb from 7.4.12 to 7.5.5

ndb config is empty / default mysql values

[24 Jan 2017 20:08] Jay Jay

ndb_2_trace.log.4

Attachment: ndb_2_trace.log.4 (application/octet-stream, text), 22.01 KiB.

[24 Jan 2017 20:11] Jay Jay

The allocating memory error message is not due to not enough ram. Even if I increase from the original 2gb ram to 64gb ram the same error happens.

[31 Jan 2017 5:15] MySQL Verification Team

Hi,

Thanks for the report, I did manage to verify this bug as described.

best regards
Bogdan