Description:
We have been running cluster successfully for about 1 year now. We crashed the database on aprox. 2010-04-27 9:05AM. After the crash, the database would not restart and had to be restored from backup.
The following query caused the database to crash:
UPDATE inventory set OrderItemID = null;
Below is the schema for the table that was being updated. After discussing with TodoInTX, it was determined that the crash happened because of slow disks, and a GCP stop occurred on both node groups, bringing the cluster down.
It appears that the cluster would not restart due to and "Incomplete log for node group: 1!" This bug may be related: http://bugs.mysql.com/bug.php?id=52156
The table has aprox. 170K rows. The cluster machines have 2gb of ram each, and overall usage on the nodes is about 12%.
See ndb_1_cluster.log.4, continued in ndb_1_cluster.log for the mangement node's log of the crash.
The most interesting error messages from the data nodes appear to be in ndb_2_error.log, and ndb_3_error.log
CREATE TABLE neatoscan.inventory(
user_name VARCHAR (255) NOT NULL,
location TEXT NOT NULL,
`source` TEXT NOT NULL,
dateAdded DATETIME NOT NULL,
sku CHAR (36),
asin VARCHAR (45) NOT NULL,
qty INT (10) UNSIGNED NOT NULL,
`condition` VARCHAR (30) NOT NULL,
itemDescription VARCHAR (1000) NOT NULL,
title VARCHAR (255) NOT NULL,
buyerEmailAddress VARCHAR (255) DEFAULT NULL,
buyerName VARCHAR (255) DEFAULT NULL,
buyerPhone VARCHAR (20) DEFAULT NULL,
addressName VARCHAR (255) DEFAULT NULL,
addressLine1 VARCHAR (255) DEFAULT NULL,
addressLine2 VARCHAR (255) DEFAULT NULL,
addressLine3 VARCHAR (255) DEFAULT NULL,
addressCity VARCHAR (255) DEFAULT NULL,
addressState VARCHAR (255) DEFAULT NULL,
addressZip VARCHAR (20) DEFAULT NULL,
addressCountryCode VARCHAR (2) DEFAULT NULL,
addressPhone VARCHAR (20) DEFAULT NULL,
addressIsConfidential TINYINT (1) DEFAULT NULL,
suppressPackingSlipChargeDetails TINYINT (1) DEFAULT NULL,
locationItem INT (10) UNSIGNED DEFAULT NULL,
inventoryStatusID BIGINT (20) UNSIGNED DEFAULT NULL,
finalPostage DECIMAL (8, 2) DEFAULT NULL,
postmarkDate DATETIME DEFAULT NULL,
freezePrice BIT (1) DEFAULT b'0',
aggressiveAmount DECIMAL (8, 2) DEFAULT NULL,
floorPrice DECIMAL (10, 2) DEFAULT NULL,
intlShipping TINYINT (1) DEFAULT NULL,
expShipping TINYINT (1) DEFAULT NULL,
relatedSKU CHAR (15) DEFAULT NULL,
warehouseNodeID VARCHAR (36) DEFAULT NULL,
amazonWeight INT (11) DEFAULT NULL,
actualWeight INT (11) DEFAULT NULL,
productCatalog ENUM ('Book', 'Music', 'Video', 'DVD', 'Video Games', 'Software', 'Toy', 'Personal Computer', 'Classical', 'Home', 'Automotive Parts and Accessories', 'Home Improvement', 'Musical Instruments', 'Wireless', 'Photography', 'Office Product', 'Watch', 'CE', 'Kitchen', 'Baby Product', 'Sports', 'UNKNOWN') DEFAULT NULL,
carrier VARCHAR (5) DEFAULT NULL,
shipMethod VARCHAR (20) DEFAULT NULL,
trackingNumber VARCHAR (30) DEFAULT NULL,
amazonFNSKU VARCHAR (20) DEFAULT NULL,
fbaShipmentID BIGINT (20) DEFAULT NULL,
listedBy VARCHAR (50) DEFAULT NULL,
oldSKU VARCHAR (50) DEFAULT NULL,
orderItemID BIGINT (20) DEFAULT NULL,
originalCode VARCHAR (30) DEFAULT NULL,
sentFollowup TINYINT (1) DEFAULT NULL,
acquisitionCost DECIMAL (10, 2) DEFAULT NULL,
PRIMARY KEY (sku),
INDEX Index_DateAdded USING BTREE (user_name, inventoryStatusID, dateAdded),
INDEX Index_InventoryStatusID USING BTREE (user_name, inventoryStatusID),
INDEX Index_oldSKU USING BTREE (oldSKU),
INDEX index_title USING BTREE (user_name, title),
INDEX Index_User_Name USING BTREE (user_name)
)
ENGINE = NDBCLUSTER
AVG_ROW_LENGTH = 188
CHARACTER SET latin1
COLLATE latin1_swedish_ci;
How to repeat:
Probably pretty hard, but in a nutshell:
1) Load a bunch of data into the inventory table
2) Run: UPDATE inventory set OrderItemID = null;
3) Some how simulate slow disks to cause a GCP stop
