Bug #46295 Create index can create crash if running out of table records
| Submitted: | 20 Jul 2009 2:47 | Modified: | 10 Aug 2009 11:19 |
|---|---|---|---|
| Reporter: | Clint Alexander | Email Updates: | |
| Status: | Closed | Impact on me: | |
| Category: | MySQL Cluster: Cluster (NDB) storage engine | Severity: | S2 (Serious) |
| Version: | mysql-5.1.34 ndb-7.0.6 | OS: | Linux |
| Assigned to: | Jonas Oreland | CPU Architecture: | Any |
| Tags: | assertion, cluster, errno: 4247 | ||
[21 Jul 2009 5:31]
Clint Alexander
Some more relevent information:
[shell]# uname -a
Linux ndbd01.in-my.net 2.6.18-128.1.16.el5 #1 SMP Tue Jun 30 06:07:26 EDT 2009 x86_64 x86_64 x86_64 GNU/Linux
I also wrote a php script (don't ask...) to recreate the error here and pull out the "SHOW WARNINGS", which I did not include in the original report.
SHOW WARNINGS:
Error 1296
Got error 4247 'Illegal index/trigger create/drop/alter request' from NDB
Error 1005
Can't create table 'aa_search.phpbb_search_matches_2540' (errno: 4247)
Error 1296
Got error 4025 'Node failure caused abort of transaction' from ndb. Releasing global schema lock
Here is that script in the event that it helps...
------------------------------
#!/usr/bin/php -f
<?php
$dbhost="localhost";
$dbuser="root";
$dbpass="";
$dbname="some_database";
$conn = mysql_connect ($dbhost, $dbuser);
if (!$conn) {
print "Unable to connect to Database\n";
exit;
}
$db = mysql_select_db($dbname, $conn);
if (!$db) {
print "Unable to select datbase $dbname\n";
exit;
}
for ($i=1; $i<=2890; $i++) {
// this should die on or around 2540
$table_name = "search_matches_".$i;
$query = "
CREATE TABLE IF NOT EXISTS `".$table_name."` (
`col1` int(11) NOT NULL auto_increment,
`col2` int(11) NOT NULL,
`col3` int(11) NOT NULL,
`col4` int(11) NOT NULL,
`col5` int(11) NOT NULL,
`col6` int(11) NOT NULL,
`col7` int(11) NOT NULL,
`col8` int(11) NOT NULL,
PRIMARY KEY (`col1`),
KEY `col2` (`col2`),
KEY `col2_2` (`col2`,`col3`),
KEY `col2_3` (`col2`,`col4`),
KEY `col2_4` (`col2`,`col5`),
KEY `col2_5` (`col2`,`col6`),
KEY `col2_6` (`col2`,`col7`)
) ENGINE=NDBCLUSTER;";
$result = mysql_query ($query, $conn);
if (mysql_error()) {
print "While creating table $table_name:\n";
print "MySQL Error: " . mysql_error() . "\n";
$query = "show warnings;";
$result = mysql_query ($query, $conn);
while ($row = mysql_fetch_array($result)) {
echo "Error Level: ".$row["Code"]
echo "Error Message: ".$row["Message"]
echo "\n";
}
break;
}
}
[21 Jul 2009 6:58]
Clint Alexander
After node fails, an attempt to do an --initial start takes over an hour during which time I could not create a table on any of the 4 SQL node: mysql> use db1; mysql> create table `testme` ( `text` varchar (100) ) ENGINE=NDBCLUSTER; ERROR 1005 (HY000): Can't create table 'db1.testme' (errno: 711) mysql>
[21 Jul 2009 7:05]
Clint Alexander
I noticed the WHILE loop at the bottom of my php script didn't copy correctly (end of lines are cropped)...
Here's a replacement
while ($row = mysql_fetch_array($result)) {
echo "Error Level: ".$row["Code"]."\n";
echo "Error Message: ".$row["Message"]."\n";
echo "\n";
}
[27 Jul 2009 9:59]
Sveta Smirnova
Thank you for the report. Verified almost as described with exception I get error 787 instead of 4247. To repeat having 4 NDB nodes are required.
[30 Jul 2009 9:42]
Bugs System
Same OS version, same MySQL Cluster version, same bug: Forced node shutdown completed. Caused by error 2301: 'Assertion(Internal error, programming error or missing error message, please report a bug). Temporary error, restart node' ...after few table creation...
[10 Aug 2009 9:01]
Jonas Oreland
7.0.7
[10 Aug 2009 11:19]
Jon Stephens
Documented bugfix in the NDB-7.0.7 changelog as follows:
Creating an index when the cluster had run out of table
records could cause mysqld to crash.
[17 Aug 2009 14:08]
Jon Stephens
Corrected changelog entry per Jonas' comment.
Portions of this website are copyright © 2001, 2002 The PHP Group
Page generated in 0.060 sec. using MySQL 8.4.7-u6-cloud
Timestamp references displayed by the system are UTC.
Content reproduced on this site is the property of the
respective copyright holders.
It is not reviewed in advance
by Oracle and does not necessarily represent the opinion of
Oracle or any other party.
Description: ------------------------------------ CLUSTER SETUP ndb_mgm> show Connected to Management Server at: 192.168.0.53:1186 Cluster Configuration --------------------- [ndbd(NDB)] 4 node(s) id=2 @192.168.0.1 (mysql-5.1.34 ndb-7.0.6, Nodegroup: 0) id=3 @192.168.0.2 (mysql-5.1.34 ndb-7.0.6, Nodegroup: 0) id=4 @192.168.0.3 (mysql-5.1.34 ndb-7.0.6, Nodegroup: 1, Master) id=5 @192.168.0.4 (mysql-5.1.34 ndb-7.0.6, Nodegroup: 1) [ndb_mgmd(MGM)] 1 node(s) id=1 @192.168.0.100 (mysql-5.1.34 ndb-7.0.6) [mysqld(API)] 9 node(s) id=11 @192.168.0.11 (mysql-5.1.34 ndb-7.0.6) id=12 @192.168.0.12 (mysql-5.1.34 ndb-7.0.6) id=13 @192.168.0.13 (mysql-5.1.34 ndb-7.0.6) id=14 @192.168.0.14 (mysql-5.1.34 ndb-7.0.6) ------------------------------------ SERVER(S) SPECS Dual Quad-Core Intel Xeon x5460 @ 3.16GHz OS: Linux CentOS 5.3 (Final) x86_64 Data Nodes: 16GB RAM SQL Nodes: 8GB RAM MGM Node : 2GB RAM ------------------------------------ DESCRIPTION I'm trying to import 2890 tables from a mysqldump file (just creating tables, no data) (On the MySQL Node...) [shell@sql01]# mysql < create_tables.sql Note, this is the equivalent of the following loop: for ($i=1; $i<=2890; $i++) { echo "Creating $i of 2890..." CREATE TABLE `mydb_search_matches_$i` ( `a_id` int(11) NOT NULL AUTO_INCREMENT, `b_id` int(11) NOT NULL, `c_id` int(11) NOT NULL, `d_id` int(11) NOT NULL, `e_id` int(11) NOT NULL, `f_id` int(11) NOT NULL, `g_id` int(11) NOT NULL, `h_id` int(11) NOT NULL, PRIMARY KEY (`a_id`), KEY `word_id` (`b_id`), KEY `word_id_2` (`b_id`,`c_id`), KEY `word_id_3` (`b_id`,`d_id`), KEY `word_id_4` (`b_id`,`e_id`), KEY `word_id_5` (`b_id`,`f_id`), KEY `word_id_6` (`b_id`,`g_id`) ) ENGINE=NDBCLUSTER; } After 2539 tables have been inserted... [shell@sql01]# mysql < create_tables.sql ERROR 1005 (HY000) at line 45707: Can't create table 'db1.mydb_search_matches_2540' (errno: 4247) [shell@sql01]# So, I start looking through all node log files. (On the MGMT Node...) [shell@mgmd]# tail /var/lib/mysql-cluster/ndb_1_cluster.log 2009-07-19 22:08:57 [MgmSrvr] ALERT -- Node 3: Forced node shutdown completed. Caused by error 2301: 'Assertion(Internal error, programming error or missing error message, please report a bug). Temporary error, restart node'. (On the MySQL Node...) [shell@sql01]# tail /var/lib/mysql/hostname.err 090719 22:08:57 [Warning] NDB: Releasing global schema lock (286)Node failure caused abort of transaction (On the Data Node...) [shell@ndbmtd03]# tail /var/lib/mysql-cluster/ndb_3_error.log Time: Sunday 19 July 2009 - 22:08:56 Status: Temporary error, restart node Message: Assertion (Internal error, programming error or missing error message, please report a bug) Error: 2301 Error data: ArrayPool<T>::getPtr Error object: ../../../../../storage/ndb/src/kernel/vm/ArrayPool.hpp line: 419 (block: DBDICT) Program: /usr/bin/ndbmtd Pid: 4316 Trace: /var/lib/mysql-cluster/ndb_3_trace.log.3 /var/lib/mysql-cluster/ndb_3_trace.log.3_t1 /var/lib/mysql-cluster/ndb_3_trace.log.3_t2 /var/lib/my ------------------------------------ CONFIG.INI [TCP DEFAULT] [NDB_MGMD DEFAULT] PortNumber = 1186 Datadir = /var/lib/mysql-cluster [NDB_MGMD] Id = 1 Hostname = 192.168.0.100 [NDBD DEFAULT] NoOfReplicas = 2 Datadir = /var/lib/mysql-cluster DataMemory = 11811160064 # 11G IndexMemory = 2147483648 # 2G StartPartialTimeout = 0 StartPartitionedTimeout = 0 StartFailureTimeout = 0 MaxNoOfConcurrentTransactions = 4096 MaxNoOfConcurrentOperations = 32768 MaxNoOfLocalOperations = 36044 StringMemory = 20 MaxNoOfTables = 20320 MaxNoOfOrderedIndexes = 40000 MaxNoOfUniqueHashIndexes = 28000 MaxNoOfAttributes = 80000 MaxNoOfTriggers = 200000 DiskCheckpointSpeedInRestart = 100M FragmentLogFileSize = 256M InitFragmentLogFiles = FULL NoOfFragmentLogFiles = 64 RedoBuffer = 32M TimeBetweenLocalCheckpoints = 24 TimeBetweenGlobalCheckpoints = 30000 LogLevelStartup = 15 LogLevelShutdown = 15 LogLevelCheckpoint = 8 LogLevelNodeRestart = 15 MaxNoOfExecutionThreads = 4 [NDBD] Id = 2 Hostname = 192.168.0.1 [NDBD] Id = 3 Hostname = 192.168.0.2 [NDBD] Id = 4 Hostname = 192.168.0.3 [NDBD] Id = 5 Hostname = 192.168.0.4 [MYSQLD DEFAULT] [MYSQLD] Id = 11 Hostname = 192.168.0.11 [MYSQLD] Id = 12 Hostname = 192.168.0.12 [MYSQLD] Id = 13 Hostname = 192.168.0.13 [MYSQLD] Id = 14 Hostname = 192.168.0.14 How to repeat: Assign config.ini as per description. Create 2540 described as per description You should see the error. Suggested fix: Unknown