| Bug #46295 | Create index can create crash if running out of table records | ||
|---|---|---|---|
| Submitted: | 20 Jul 2009 2:47 | Modified: | 10 Aug 2009 11:19 |
| Reporter: | Clint Alexander | Email Updates: | |
| Status: | Closed | Impact on me: | |
| Category: | MySQL Cluster: Cluster (NDB) storage engine | Severity: | S2 (Serious) |
| Version: | mysql-5.1.34 ndb-7.0.6 | OS: | Linux |
| Assigned to: | Jonas Oreland | CPU Architecture: | Any |
| Tags: | assertion, cluster, errno: 4247 | ||
[21 Jul 2009 5:31]
Clint Alexander
Some more relevent information:
[shell]# uname -a
Linux ndbd01.in-my.net 2.6.18-128.1.16.el5 #1 SMP Tue Jun 30 06:07:26 EDT 2009 x86_64 x86_64 x86_64 GNU/Linux
I also wrote a php script (don't ask...) to recreate the error here and pull out the "SHOW WARNINGS", which I did not include in the original report.
SHOW WARNINGS:
Error 1296
Got error 4247 'Illegal index/trigger create/drop/alter request' from NDB
Error 1005
Can't create table 'aa_search.phpbb_search_matches_2540' (errno: 4247)
Error 1296
Got error 4025 'Node failure caused abort of transaction' from ndb. Releasing global schema lock
Here is that script in the event that it helps...
------------------------------
#!/usr/bin/php -f
<?php
$dbhost="localhost";
$dbuser="root";
$dbpass="";
$dbname="some_database";
$conn = mysql_connect ($dbhost, $dbuser);
if (!$conn) {
print "Unable to connect to Database\n";
exit;
}
$db = mysql_select_db($dbname, $conn);
if (!$db) {
print "Unable to select datbase $dbname\n";
exit;
}
for ($i=1; $i<=2890; $i++) {
// this should die on or around 2540
$table_name = "search_matches_".$i;
$query = "
CREATE TABLE IF NOT EXISTS `".$table_name."` (
`col1` int(11) NOT NULL auto_increment,
`col2` int(11) NOT NULL,
`col3` int(11) NOT NULL,
`col4` int(11) NOT NULL,
`col5` int(11) NOT NULL,
`col6` int(11) NOT NULL,
`col7` int(11) NOT NULL,
`col8` int(11) NOT NULL,
PRIMARY KEY (`col1`),
KEY `col2` (`col2`),
KEY `col2_2` (`col2`,`col3`),
KEY `col2_3` (`col2`,`col4`),
KEY `col2_4` (`col2`,`col5`),
KEY `col2_5` (`col2`,`col6`),
KEY `col2_6` (`col2`,`col7`)
) ENGINE=NDBCLUSTER;";
$result = mysql_query ($query, $conn);
if (mysql_error()) {
print "While creating table $table_name:\n";
print "MySQL Error: " . mysql_error() . "\n";
$query = "show warnings;";
$result = mysql_query ($query, $conn);
while ($row = mysql_fetch_array($result)) {
echo "Error Level: ".$row["Code"]
echo "Error Message: ".$row["Message"]
echo "\n";
}
break;
}
}
[21 Jul 2009 6:58]
Clint Alexander
After node fails, an attempt to do an --initial start takes over an hour during which time I could not create a table on any of the 4 SQL node: mysql> use db1; mysql> create table `testme` ( `text` varchar (100) ) ENGINE=NDBCLUSTER; ERROR 1005 (HY000): Can't create table 'db1.testme' (errno: 711) mysql>
[21 Jul 2009 7:05]
Clint Alexander
I noticed the WHILE loop at the bottom of my php script didn't copy correctly (end of lines are cropped)...
Here's a replacement
while ($row = mysql_fetch_array($result)) {
echo "Error Level: ".$row["Code"]."\n";
echo "Error Message: ".$row["Message"]."\n";
echo "\n";
}
[27 Jul 2009 9:59]
Sveta Smirnova
Thank you for the report. Verified almost as described with exception I get error 787 instead of 4247. To repeat having 4 NDB nodes are required.
[30 Jul 2009 9:42]
Bugs System
Same OS version, same MySQL Cluster version, same bug: Forced node shutdown completed. Caused by error 2301: 'Assertion(Internal error, programming error or missing error message, please report a bug). Temporary error, restart node' ...after few table creation...
[10 Aug 2009 9:01]
Jonas Oreland
7.0.7
[10 Aug 2009 11:19]
Jon Stephens
Documented bugfix in the NDB-7.0.7 changelog as follows:
Creating an index when the cluster had run out of table
records could cause mysqld to crash.
[17 Aug 2009 14:08]
Jon Stephens
Corrected changelog entry per Jonas' comment.
Description: ------------------------------------ CLUSTER SETUP ndb_mgm> show Connected to Management Server at: 192.168.0.53:1186 Cluster Configuration --------------------- [ndbd(NDB)] 4 node(s) id=2 @192.168.0.1 (mysql-5.1.34 ndb-7.0.6, Nodegroup: 0) id=3 @192.168.0.2 (mysql-5.1.34 ndb-7.0.6, Nodegroup: 0) id=4 @192.168.0.3 (mysql-5.1.34 ndb-7.0.6, Nodegroup: 1, Master) id=5 @192.168.0.4 (mysql-5.1.34 ndb-7.0.6, Nodegroup: 1) [ndb_mgmd(MGM)] 1 node(s) id=1 @192.168.0.100 (mysql-5.1.34 ndb-7.0.6) [mysqld(API)] 9 node(s) id=11 @192.168.0.11 (mysql-5.1.34 ndb-7.0.6) id=12 @192.168.0.12 (mysql-5.1.34 ndb-7.0.6) id=13 @192.168.0.13 (mysql-5.1.34 ndb-7.0.6) id=14 @192.168.0.14 (mysql-5.1.34 ndb-7.0.6) ------------------------------------ SERVER(S) SPECS Dual Quad-Core Intel Xeon x5460 @ 3.16GHz OS: Linux CentOS 5.3 (Final) x86_64 Data Nodes: 16GB RAM SQL Nodes: 8GB RAM MGM Node : 2GB RAM ------------------------------------ DESCRIPTION I'm trying to import 2890 tables from a mysqldump file (just creating tables, no data) (On the MySQL Node...) [shell@sql01]# mysql < create_tables.sql Note, this is the equivalent of the following loop: for ($i=1; $i<=2890; $i++) { echo "Creating $i of 2890..." CREATE TABLE `mydb_search_matches_$i` ( `a_id` int(11) NOT NULL AUTO_INCREMENT, `b_id` int(11) NOT NULL, `c_id` int(11) NOT NULL, `d_id` int(11) NOT NULL, `e_id` int(11) NOT NULL, `f_id` int(11) NOT NULL, `g_id` int(11) NOT NULL, `h_id` int(11) NOT NULL, PRIMARY KEY (`a_id`), KEY `word_id` (`b_id`), KEY `word_id_2` (`b_id`,`c_id`), KEY `word_id_3` (`b_id`,`d_id`), KEY `word_id_4` (`b_id`,`e_id`), KEY `word_id_5` (`b_id`,`f_id`), KEY `word_id_6` (`b_id`,`g_id`) ) ENGINE=NDBCLUSTER; } After 2539 tables have been inserted... [shell@sql01]# mysql < create_tables.sql ERROR 1005 (HY000) at line 45707: Can't create table 'db1.mydb_search_matches_2540' (errno: 4247) [shell@sql01]# So, I start looking through all node log files. (On the MGMT Node...) [shell@mgmd]# tail /var/lib/mysql-cluster/ndb_1_cluster.log 2009-07-19 22:08:57 [MgmSrvr] ALERT -- Node 3: Forced node shutdown completed. Caused by error 2301: 'Assertion(Internal error, programming error or missing error message, please report a bug). Temporary error, restart node'. (On the MySQL Node...) [shell@sql01]# tail /var/lib/mysql/hostname.err 090719 22:08:57 [Warning] NDB: Releasing global schema lock (286)Node failure caused abort of transaction (On the Data Node...) [shell@ndbmtd03]# tail /var/lib/mysql-cluster/ndb_3_error.log Time: Sunday 19 July 2009 - 22:08:56 Status: Temporary error, restart node Message: Assertion (Internal error, programming error or missing error message, please report a bug) Error: 2301 Error data: ArrayPool<T>::getPtr Error object: ../../../../../storage/ndb/src/kernel/vm/ArrayPool.hpp line: 419 (block: DBDICT) Program: /usr/bin/ndbmtd Pid: 4316 Trace: /var/lib/mysql-cluster/ndb_3_trace.log.3 /var/lib/mysql-cluster/ndb_3_trace.log.3_t1 /var/lib/mysql-cluster/ndb_3_trace.log.3_t2 /var/lib/my ------------------------------------ CONFIG.INI [TCP DEFAULT] [NDB_MGMD DEFAULT] PortNumber = 1186 Datadir = /var/lib/mysql-cluster [NDB_MGMD] Id = 1 Hostname = 192.168.0.100 [NDBD DEFAULT] NoOfReplicas = 2 Datadir = /var/lib/mysql-cluster DataMemory = 11811160064 # 11G IndexMemory = 2147483648 # 2G StartPartialTimeout = 0 StartPartitionedTimeout = 0 StartFailureTimeout = 0 MaxNoOfConcurrentTransactions = 4096 MaxNoOfConcurrentOperations = 32768 MaxNoOfLocalOperations = 36044 StringMemory = 20 MaxNoOfTables = 20320 MaxNoOfOrderedIndexes = 40000 MaxNoOfUniqueHashIndexes = 28000 MaxNoOfAttributes = 80000 MaxNoOfTriggers = 200000 DiskCheckpointSpeedInRestart = 100M FragmentLogFileSize = 256M InitFragmentLogFiles = FULL NoOfFragmentLogFiles = 64 RedoBuffer = 32M TimeBetweenLocalCheckpoints = 24 TimeBetweenGlobalCheckpoints = 30000 LogLevelStartup = 15 LogLevelShutdown = 15 LogLevelCheckpoint = 8 LogLevelNodeRestart = 15 MaxNoOfExecutionThreads = 4 [NDBD] Id = 2 Hostname = 192.168.0.1 [NDBD] Id = 3 Hostname = 192.168.0.2 [NDBD] Id = 4 Hostname = 192.168.0.3 [NDBD] Id = 5 Hostname = 192.168.0.4 [MYSQLD DEFAULT] [MYSQLD] Id = 11 Hostname = 192.168.0.11 [MYSQLD] Id = 12 Hostname = 192.168.0.12 [MYSQLD] Id = 13 Hostname = 192.168.0.13 [MYSQLD] Id = 14 Hostname = 192.168.0.14 How to repeat: Assign config.ini as per description. Create 2540 described as per description You should see the error. Suggested fix: Unknown