MySQL Bugs: #52055: NDB node shutdown and NDB misbehavior version 7.0.7

Bug #52055	NDB node shutdown and NDB misbehavior version 7.0.7
Submitted:	15 Mar 2010 8:34	Modified:	24 May 2010 20:13
Reporter:	Rosario Arun	Email Updates:
Status:	Closed	Impact on me:	None
Category:	MySQL Cluster: Cluster (NDB) storage engine	Severity:	S1 (Critical)
Version:	mysql-5.1-telco-7.0	OS:	Linux (SUSE Enterprise Linux)
Assigned to:	Jonas Oreland	CPU Architecture:	Any
Tags:	database import issue version 7.0.7, ndbd node shutdown

Description:
When I do some database import from an SQL dump, or sometimes on its own the NDB data nodes shutsdown automatically. 

Most of the time when I try to import from the sql dump i get 707 Error. I know this means max no of table. I have only 5 tables in the database and still i get this error. 

Here is the logs of  ndb_*_error.log

Status: Temporary error, restart node
Message: Assertion (Internal error, programming error or missing error message, please report a bug)
Error: 2301
Error data: ArrayPool<T>::getPtr
Error object: ../../../../../storage/ndb/src/kernel/vm/ArrayPool.hpp line: 422 (block: DBDICT)
Program: /usr/sbin/ndbd
Pid: 4217
Trace: /var/lib/mysql-cluster//ndb_2_trace.log.2
Version: mysql-5.1.35 ndb-7.0.7

The bug, http://bugs.mysql.com/bug.php?id=46295 tells that this issue is resolved in 7.0.7 version. I am using 7.0.7 version. 

Some other misbehaviours of the NDB, 
When I truncate a table it gets dropped and when i try to create it again it says table already exsists. 

All report mem, output
Both the data node 22 % data usage and 13 percent % index usage.

Thanks in advance

How to repeat:
In the mysql-5.1.35 ndb-7.0.7 version. Try to table a mysql dump from an existing database and change the engine to ndbcluster in the create table statements and try to import it into the ndbcluster. You will get error like 707 error. If you try again an again two three times your ndbd will shutdown automatically. 

And for the ndbd to come up again it would take more than half an hour.

Hi Rosario,

Could you please provide the full ndb_error_reporter output as described in:

 http://dev.mysql.com/doc/refman/5.1/en/mysql-cluster-programs-ndb-error-reporter.html

If you upload it to our ftp server please put 52055 somewhere in the filename.

/Gustaf

ndb_error_reporter

Attachment: 52055__ndb_error_report_20100318180120.tar.bz2 (application/octet-stream, text), 154.58 KiB.

Error log or all the nodes Complete version

Attachment: 52055_ALLNODES_ndb_error_report_20100318181802.tar.bz2 (application/octet-stream, text), 256.85 KiB.

Hi Gustaf Thorslund ,
Thanks for your response. I have uploaded the error dump from ndb_error_reporter. The second file 52055_ALLNODES_ndb_error_report_20100318181802.tar.bz2 holds the complete dump. 

Thanks in advance.

Hi,

1) Can you please try with a newer version, e.g 7.0.14
or 7.1.3

2) If this still fails, can you please upload the schema definition.

3) Also, have you upgrade from 6.3 ?

/Jonas

This is repeatable with MySQL Cluster 7.1.3 (so I guess also with 7.0.14).

* Creating 300 tables, failed with out of attributes
* Increasing MaxNoAttributes to 10000
* Failure after 107 where created, error 2301
* Cleaning up everything i.e. removing all ndb related files and MySQL's datadir
* Start again: error 136 No more room in index file
* Reconfiguring to give more index memory
* Again, same error 2301 with trace here below

Configuration:

[ndbd default]
NoOfReplicas = 1
datamemory = 200M
indexmemory = 70M
datadir = /opt/mysql/clusters/machA/ndb

MaxNoOfAttributes=10000

MaxNoOfConcurrentOperations=500000
MaxNoOfConcurrentTransactions=20000

[ndb_mgmd default]
datadir = /opt/mysql/clusters/machA/ndb

[ndb_mgmd]
hostname = localhost
PortNumber = 1186
id = 1

[ndbd]
hostname = localhost
id = 2

[api]
id = 20
[api]
id = 21
[api]
id = 22

Attaching file which has error and trace.

Just a FYI, the Python script creating a rather basic table 319 times:

import mysql.connector as db

def main():
  
  cnx = db.connect(user='root',db='test')
  cur = cnx.cursor()
  
  create = "CREATE TABLE t%d (id INT UNSIGNED NOT NULL AUTO_INCREMENT,"\
    " c1 INT, c2 INT, c3 VARCHAR(400), c4 VARCHAR(200), c5 INT, c6 BIT,"\
    "PRIMARY KEY (id)) ENGINE=ndb"
  
  for i in xrange(1,320):
    cur.execute("DROP TABLE IF EXISTS t%d" % i)
    cur.execute(create % i)
  
  cur.close()
  cnx.close()

Yes, repeatable:
1) Make sure you create enough tables so error "error code 136: No more room in index file" is generated. This can be done with the above configuration
2) Restart Cluster
3) Try again recreating these tables.
4) Data Node shuts down.

Seems not to be related to setting MaxNoOfOrderedIndexes higher (defaults to 128). Setting it to 500 crashes the data node as well, at creation of table t217 or t128.

A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/108999

3544 Jonas Oreland	2010-05-24
      ndb - bug#52055 - fix problem where index could get too high id, causing assertion

pushed to 7.0.15 and 7.1.4
DOCS: when creating an index, code failed to check that id allocate was within
  range of possible id's for an index, causing assertion

Documented bugfix in the NDB-7.0.15 and 7.1.4 changelogs as follows:

        When creating an index, NDB failed to check whether the internal
        ID allocated to the index was within the allowable range,
        leading to an assertion. This issue could manifest itself as a
        data node failure with NDB error 707 (No more table metadata
        records (increase MaxNoOfTables)), when creating tables in rapid
        succession (for example, by a script, or when importing from
        mysqldump), even with a relatively high value for MaxNoOfTables
        and a relatively low number of tables.

Closed.