Bug #81287 ndbmtd crashes
Submitted: 3 May 2016 3:09 Modified: 4 Jun 2016 15:02
Reporter: Hieu Dinh Email Updates:
Status: No Feedback Impact on me:
None 
Category:MySQL Cluster: Cluster (NDB) storage engine Severity:S1 (Critical)
Version:mysql-5.6.28 ndb-7.4.10 OS:Debian
Assigned to: MySQL Verification Team CPU Architecture:Any

[3 May 2016 3:09] Hieu Dinh
Description:
root@db2:~# tail /usr/local/mysql/ndbd_data/ndb_5_error.log
Time: Monday 2 May 2016 - 17:36:05
Status: Temporary error, restart node
Message: WatchDog terminate, internal error or massive overload on the machine running this node (Internal error, programming error or missing error message, please report a bug)
Error: 6050
Error data: Job Handling
Error object: /export/home/pb2/build/sb_0-17731890-1453887053.27/mysql-cluster-gpl-7.4.10/storage/ndb/src/kernel/vm/WatchDog.cpp
Program: ndbmtd
Pid: 5326
Version: mysql-5.6.28 ndb-7.4.10
Trace: /usr/local/mys

How to repeat:
On management nodes (192.168.100.11, 192.168.100.12, 192.168.100.13)

root@192.168.100.11:~
$ cat /var/lib/mysql-cluster/config.ini
[ndbd default]
# Options affecting ndbd processes on all data nodes:
NoOfReplicas=2    # Number of replicas
DataMemory=29G    # How much memory to allocate for data storage
IndexMemory=29G   # How much memory to allocate for index storage
                  # For DataMemory and IndexMemory, we have used the
                  # default values. Since the "world" database takes up
                  # only about 500KB, this should be more than enough for
                  # this example Cluster setup.

MaxNoOfExecutionThreads = 4
DataDir=/usr/local/mysql/ndbd_data
FileSystemPathUndoFiles=/datadisk/mysql-cluster/dnlogs
FileSystemPathDataFiles=/datadisk/mysql-cluster/dndata

[tcp default]
# TCP/IP options:
portnumber=2202   # This the default; however, you can use any
                  # port that is free for all the hosts in the cluster
                  # Note: It is recommended that you do not specify the port
                  # number at all and simply allow the default value to be used
                  # instead

# one [ndb_mgmd] section per management node
[ndb_mgmd]
# Management process options:
hostname=192.168.100.11           # Hostname or IP address of MGM node
datadir=/var/lib/mysql-cluster  # Directory for MGM node log files

[ndb_mgmd]
# Management process options:
hostname=192.168.100.12           # Hostname or IP address of MGM node
datadir=/var/lib/mysql-cluster  # Directory for MGM node log files

[ndb_mgmd]
# Management process options:
hostname=192.168.100.13           # Hostname or IP address of MGM node
datadir=/var/lib/mysql-cluster  # Directory for MGM node log files

# one [ndbd] section per data node
[ndbd]
hostname=192.168.100.51           # Hostname or IP address
#datadir=/usr/local/mysql/ndbd_data   # Directory for this data node's data files

[ndbd]
hostname=192.168.100.52           # Hostname or IP address
#datadir=/usr/local/mysql/ndbd_data   # Directory for this data node's data files

# one [mysqld] section per SQL node
[mysqld]
# SQL node options:
hostname=192.168.100.11            # Hostname or IP address
                                # (additional mysqld connections can be
                                # specified for this node for various
                                # purposes such as running ndb_restore)

[mysqld]
# SQL node options:
hostname=192.168.100.12            # Hostname or IP address

[mysqld]
# SQL node options:
hostname=192.168.100.13            # Hostname or IP address

root@192.168.100.11:~
/opt/mysql/server-5.6/bin/ndb_mgmd -f /var/lib/mysql-cluster/config.ini

On data nodes (192.168.100.51, 192.168.100.52)

root@192.168.100.51:~
cat /etc/my.cnf
[mysqld]
# Options for mysqld process:
ndbcluster                      # run NDB storage engine

[mysql_cluster]
# Options for MySQL Cluster processes:
# Location of management server
# ndb-connectstring=192.168.100.11
ndb-connectstring=192.168.100.11,192.168.100.12,192.168.100.13

root@192.168.100.51:~
/opt/mysql/server-5.6/bin/ndbmtd
[4 May 2016 15:02] MySQL Verification Team
Hi,

Did you check the CPU and IO usage on your servers at the time this problem occurred? Any monitoring system that can show you cpu and io usage at time of the problem?

Without full log files we really can only guess what's going on. Please use ndb_error_reporter to collect all log files.

kind regards
Bogdan Kecman
[5 Jun 2016 1:00] Bugs System
No feedback was provided for this bug for over a month, so it is
being suspended automatically. If you are able to provide the
information that was originally requested, please do so and change
the status of the bug back to "Open".