Description:
when trying to run a batch insert to fill a table with data in MySQL cluster database 7.2.10 ,suddenly all data nodes in the cluster are forcing shutdown, and I get the error below(error repeated several times):
------------------------------
Forced node shutdown completed. Caused by error 1217 ,No message slogan found (please report a bug if you get this error code) (Unknown)
-----------------------------------
I have been monitoring the dataMemory and indexMemory and it were fine, last reading were:
-------------------------
Node 3: Data usage is 6%
Node 3: Index usage is 4%
Node 4: Data usage is 6%
Node 4: Index usage is 4%
-----------------------------------
Here's a part of my management log:
-------------------------------------
2013-04-30 09:00:29 [MgmtSrvr] INFO -- Node 4: Local checkpoint 1785 started. Keep GCI = 2924750 oldest restorable GCI = 2924751
2013-04-30 09:02:10 [MgmtSrvr] INFO -- Node 4: Local checkpoint 1785 completed
2013-04-30 09:02:11 [MgmtSrvr] INFO -- Node 4: Local checkpoint 1786 started. Keep GCI = 2924799 oldest restorable GCI = 2924800
2013-04-30 09:03:52 [MgmtSrvr] INFO -- Node 4: Local checkpoint 1786 completed
2013-04-30 09:03:53 [MgmtSrvr] INFO -- Node 4: Local checkpoint 1787 started. Keep GCI = 2924848 oldest restorable GCI = 2924849
2013-04-30 09:05:34 [MgmtSrvr] INFO -- Node 4: Local checkpoint 1787 completed
2013-04-30 09:05:35 [MgmtSrvr] INFO -- Node 4: Local checkpoint 1788 started. Keep GCI = 2924897 oldest restorable GCI = 2924898
2013-04-30 09:07:15 [MgmtSrvr] INFO -- Node 4: Local checkpoint 1788 completed
2013-04-30 09:07:16 [MgmtSrvr] INFO -- Node 4: Local checkpoint 1789 started. Keep GCI = 2924946 oldest restorable GCI = 2924947
2013-04-30 09:08:57 [MgmtSrvr] INFO -- Node 4: Local checkpoint 1789 completed
2013-04-30 09:08:58 [MgmtSrvr] INFO -- Node 4: Local checkpoint 1790 started. Keep GCI = 2924995 oldest restorable GCI = 2924996
2013-04-30 09:10:39 [MgmtSrvr] INFO -- Node 4: Local checkpoint 1790 completed
2013-04-30 09:10:40 [MgmtSrvr] INFO -- Node 4: Local checkpoint 1791 started. Keep GCI = 2925044 oldest restorable GCI = 2925045
2013-04-30 09:11:29 [MgmtSrvr] ALERT -- Node 1: Node 3 Disconnected
2013-04-30 09:11:29 [MgmtSrvr] ALERT -- Node 1: Node 4 Disconnected
2013-04-30 09:11:29 [MgmtSrvr] ALERT -- Node 4: Forced node shutdown completed. Caused by error 1217: 'No message slogan found (please report a bug if you get this error code)(Unknown). Unknown'.
2013-04-30 09:11:29 [MgmtSrvr] ALERT -- Node 3: Forced node shutdown completed. Caused by error 1217: 'No message slogan found (please report a bug if you get this error code)(Unknown). Unknown'.
-------------------------------------
and Here's my config.ini file:
---------------------------------
[ndbd default]
# Options affecting ndbd processes on all data nodes:
NoOfReplicas=2 # Number of replicas
DataDir=C:/mysql/bin/cluster-data # Directory for each data node's data files
# Forward slashes used in directory path,
# rather than backslashes. This is correct;
# see Important note in text
DataMemory=7G # Memory allocated to data storage
IndexMemory=400M # Memory allocated to index storage
# For DataMemory and IndexMemory, we have used the
# default values. Since the "world" database takes up
# only about 500KB, this should be more than enough for
# this example Cluster setup.
MaxNoOfAttributes = 10000
MaxNoOfOrderedIndexes = 1024
MaxNoOfConcurrentTransactions = 500000
MaxNoOfConcurrentOperations = 500000
MaxNoOfLocalOperations = 550000
[ndb_mgmd]
# Management process options:
nodeId=1
HostName=10.245.2.10 # Hostname or IP address of management node
DataDir=C:/mysql/bin/cluster-logs # Directory for management node log files
[ndb_mgmd]
# Management process options:
nodeId=2
HostName=10.245.2.11 # Hostname or IP address of management node
DataDir=C:/mysql/bin/cluster-logs # Directory for management node log files
[ndbd]
# Options for data node "A":
HostName=10.245.2.14 # Hostname or IP address
[ndbd]
# Options for data node "B":
HostName=10.245.2.130 # Hostname or IP address
[mysqld]
# SQL node options:
HostName=10.245.2.10 # Hostname or IP address
[mysqld]
# SQL node options:
HostName=10.245.2.11 # Hostname or IP address
---------------------------------
Thanks in advance.
How to repeat:
1- create a new database .."hpm"
2- create a new table.In my case table name is "structres" with the following DDL:
------------------------------------------------
CREATE TABLE `structures` (
`STRUCTURE_ID` bigint(20) NOT NULL AUTO_INCREMENT,
`PARENT_STRUCTURE_ID` bigint(20) DEFAULT NULL,
`NAME_UK` varchar(250) DEFAULT NULL,
`NAME_DK` varchar(250) DEFAULT NULL,
`NAME_PL` varchar(250) DEFAULT NULL,
`NAME_SV` varchar(250) DEFAULT NULL,
`NAME_NO` varchar(250) DEFAULT NULL,
`NAME_FI` varchar(250) DEFAULT NULL,
`NAME_LT` varchar(250) DEFAULT NULL,
`DESCRIPTION_UK` text,
`DESCRIPTION_DK` text,
`DESCRIPTION_PL` text,
`DESCRIPTION_SV` text,
`DESCRIPTION_NO` text,
`DESCRIPTION_FI` text,
`DESCRIPTION_LT` text,
`IMAGE1` text,
`IMAGE2` text,
`SEQUENCE` bigint(20) DEFAULT NULL,
`CREATED_DATE` datetime DEFAULT NULL,
`CHANGED_DATE` datetime DEFAULT NULL,
`STATUS` varchar(10) DEFAULT 'INSERT',
`STRUCTURE` varchar(100) DEFAULT NULL,
`TYPE` varchar(1) DEFAULT NULL,
`ROOT_STRUCTURE_ID` bigint(20) DEFAULT NULL,
`TARGET` varchar(50) DEFAULT 'WEB',
PRIMARY KEY (`STRUCTURE_ID`)
) ENGINE=ndbcluster AUTO_INCREMENT=1363161223449 DEFAULT CHARSET=utf8
-------------------------------------------------
3- running a lot of inserts on hpm.structures table,where batch inserts 50 rows at the time.
4- after some time(around 20 minutes) you will get the error and all data nodes will force shutdown.
Description: when trying to run a batch insert to fill a table with data in MySQL cluster database 7.2.10 ,suddenly all data nodes in the cluster are forcing shutdown, and I get the error below(error repeated several times): ------------------------------ Forced node shutdown completed. Caused by error 1217 ,No message slogan found (please report a bug if you get this error code) (Unknown) ----------------------------------- I have been monitoring the dataMemory and indexMemory and it were fine, last reading were: ------------------------- Node 3: Data usage is 6% Node 3: Index usage is 4% Node 4: Data usage is 6% Node 4: Index usage is 4% ----------------------------------- Here's a part of my management log: ------------------------------------- 2013-04-30 09:00:29 [MgmtSrvr] INFO -- Node 4: Local checkpoint 1785 started. Keep GCI = 2924750 oldest restorable GCI = 2924751 2013-04-30 09:02:10 [MgmtSrvr] INFO -- Node 4: Local checkpoint 1785 completed 2013-04-30 09:02:11 [MgmtSrvr] INFO -- Node 4: Local checkpoint 1786 started. Keep GCI = 2924799 oldest restorable GCI = 2924800 2013-04-30 09:03:52 [MgmtSrvr] INFO -- Node 4: Local checkpoint 1786 completed 2013-04-30 09:03:53 [MgmtSrvr] INFO -- Node 4: Local checkpoint 1787 started. Keep GCI = 2924848 oldest restorable GCI = 2924849 2013-04-30 09:05:34 [MgmtSrvr] INFO -- Node 4: Local checkpoint 1787 completed 2013-04-30 09:05:35 [MgmtSrvr] INFO -- Node 4: Local checkpoint 1788 started. Keep GCI = 2924897 oldest restorable GCI = 2924898 2013-04-30 09:07:15 [MgmtSrvr] INFO -- Node 4: Local checkpoint 1788 completed 2013-04-30 09:07:16 [MgmtSrvr] INFO -- Node 4: Local checkpoint 1789 started. Keep GCI = 2924946 oldest restorable GCI = 2924947 2013-04-30 09:08:57 [MgmtSrvr] INFO -- Node 4: Local checkpoint 1789 completed 2013-04-30 09:08:58 [MgmtSrvr] INFO -- Node 4: Local checkpoint 1790 started. Keep GCI = 2924995 oldest restorable GCI = 2924996 2013-04-30 09:10:39 [MgmtSrvr] INFO -- Node 4: Local checkpoint 1790 completed 2013-04-30 09:10:40 [MgmtSrvr] INFO -- Node 4: Local checkpoint 1791 started. Keep GCI = 2925044 oldest restorable GCI = 2925045 2013-04-30 09:11:29 [MgmtSrvr] ALERT -- Node 1: Node 3 Disconnected 2013-04-30 09:11:29 [MgmtSrvr] ALERT -- Node 1: Node 4 Disconnected 2013-04-30 09:11:29 [MgmtSrvr] ALERT -- Node 4: Forced node shutdown completed. Caused by error 1217: 'No message slogan found (please report a bug if you get this error code)(Unknown). Unknown'. 2013-04-30 09:11:29 [MgmtSrvr] ALERT -- Node 3: Forced node shutdown completed. Caused by error 1217: 'No message slogan found (please report a bug if you get this error code)(Unknown). Unknown'. ------------------------------------- and Here's my config.ini file: --------------------------------- [ndbd default] # Options affecting ndbd processes on all data nodes: NoOfReplicas=2 # Number of replicas DataDir=C:/mysql/bin/cluster-data # Directory for each data node's data files # Forward slashes used in directory path, # rather than backslashes. This is correct; # see Important note in text DataMemory=7G # Memory allocated to data storage IndexMemory=400M # Memory allocated to index storage # For DataMemory and IndexMemory, we have used the # default values. Since the "world" database takes up # only about 500KB, this should be more than enough for # this example Cluster setup. MaxNoOfAttributes = 10000 MaxNoOfOrderedIndexes = 1024 MaxNoOfConcurrentTransactions = 500000 MaxNoOfConcurrentOperations = 500000 MaxNoOfLocalOperations = 550000 [ndb_mgmd] # Management process options: nodeId=1 HostName=10.245.2.10 # Hostname or IP address of management node DataDir=C:/mysql/bin/cluster-logs # Directory for management node log files [ndb_mgmd] # Management process options: nodeId=2 HostName=10.245.2.11 # Hostname or IP address of management node DataDir=C:/mysql/bin/cluster-logs # Directory for management node log files [ndbd] # Options for data node "A": HostName=10.245.2.14 # Hostname or IP address [ndbd] # Options for data node "B": HostName=10.245.2.130 # Hostname or IP address [mysqld] # SQL node options: HostName=10.245.2.10 # Hostname or IP address [mysqld] # SQL node options: HostName=10.245.2.11 # Hostname or IP address --------------------------------- Thanks in advance. How to repeat: 1- create a new database .."hpm" 2- create a new table.In my case table name is "structres" with the following DDL: ------------------------------------------------ CREATE TABLE `structures` ( `STRUCTURE_ID` bigint(20) NOT NULL AUTO_INCREMENT, `PARENT_STRUCTURE_ID` bigint(20) DEFAULT NULL, `NAME_UK` varchar(250) DEFAULT NULL, `NAME_DK` varchar(250) DEFAULT NULL, `NAME_PL` varchar(250) DEFAULT NULL, `NAME_SV` varchar(250) DEFAULT NULL, `NAME_NO` varchar(250) DEFAULT NULL, `NAME_FI` varchar(250) DEFAULT NULL, `NAME_LT` varchar(250) DEFAULT NULL, `DESCRIPTION_UK` text, `DESCRIPTION_DK` text, `DESCRIPTION_PL` text, `DESCRIPTION_SV` text, `DESCRIPTION_NO` text, `DESCRIPTION_FI` text, `DESCRIPTION_LT` text, `IMAGE1` text, `IMAGE2` text, `SEQUENCE` bigint(20) DEFAULT NULL, `CREATED_DATE` datetime DEFAULT NULL, `CHANGED_DATE` datetime DEFAULT NULL, `STATUS` varchar(10) DEFAULT 'INSERT', `STRUCTURE` varchar(100) DEFAULT NULL, `TYPE` varchar(1) DEFAULT NULL, `ROOT_STRUCTURE_ID` bigint(20) DEFAULT NULL, `TARGET` varchar(50) DEFAULT 'WEB', PRIMARY KEY (`STRUCTURE_ID`) ) ENGINE=ndbcluster AUTO_INCREMENT=1363161223449 DEFAULT CHARSET=utf8 ------------------------------------------------- 3- running a lot of inserts on hpm.structures table,where batch inserts 50 rows at the time. 4- after some time(around 20 minutes) you will get the error and all data nodes will force shutdown.