Bug #70363 Got error 707 'No more table metadata records (increase MaxNoOfTables)' from NDB
Submitted: 17 Sep 2013 10:48 Modified: 19 Sep 2013 12:06
Reporter: prasanth katta Email Updates:
Status: Not a Bug Impact on me:
None 
Category:MySQL Cluster: Cluster (NDB) storage engine Severity:S2 (Serious)
Version:mysql-5.6.11 ndb-7.3.2 OS:Linux (Ubuntu 12.04 LTS)
Assigned to: CPU Architecture:Any

[17 Sep 2013 10:48] prasanth katta
Description:
i am facing problem while dumpig data to mysql cluster. when ever i dump the data (.sql which is of 800MB in size with 400 tables) in to mysql cluster i am getting the following error "No more table metadata records (increase MaxNoOfTables)' from NDBCLUSTER"
2013-09-17 09:15:41 [MgmtSrvr] INFO     -- Node 3: mysqld --server-id=0
2013-09-17 09:15:42 [MgmtSrvr] INFO     -- Node 2: Node 3 Connected
2013-09-17 09:15:42 [MgmtSrvr] INFO     -- Node 2: Node 3: API mysql-5.6.11 ndb-7.3.2
2013-09-17 09:27:32 [MgmtSrvr] INFO     -- Node 2: Data usage is 0%(8 32K pages of total 524288)
2013-09-17 09:27:32 [MgmtSrvr] INFO     -- Node 2: Index usage is 0%(12 8K pages of total 1310752)
2013-09-17 09:27:32 [MgmtSrvr] INFO     -- Node 2: Resource 0 min: 524407 max: 859521 curr: 331025
2013-09-17 09:27:32 [MgmtSrvr] INFO     -- Node 2: Resource 3 min: 851976 max: 851976 curr: 327696
2013-09-17 09:27:32 [MgmtSrvr] INFO     -- Node 2: Resource 5 min: 1152 max: 1152 curr: 1088
2013-09-17 09:27:32 [MgmtSrvr] INFO     -- Node 2: Resource 7 min: 2240 max: 2240 curr: 2240
2013-09-17 09:27:32 [MgmtSrvr] INFO     -- Node 2: Resource 9 min: 64 max: 0 curr: 1

[ndb_mgmd]
NodeId=1
HostName=127.0.0.1          # This is an IP address of the cluster management node
#DataDir=/var/lib/mysql-cluster  # This is a data directory for the cluster management node
#DataDir=/usr/local/mysql/mysql-cluster
DataDir=/usr/local/mysql/data
Portnumber=1186

[tcp default]
# TCP/IP options:
SendBufferMemory=8M
ReceiveBufferMemory=8M
 
[ndbd default]
#DataDir=/var/lib/mysql-cluster
#DataDir=/usr/local/mysql/mysql-cluster
DataDir=/usr/local/mysql/data
NoOfReplicas=1
FileSystemPathDD=/var/lib/mysqlcluster/
#BackupDataDir=/var/lib/mysqlcluster/backup/
#FileSystemPathUndoFiles=/var/lib/mysqlcluster/
#FileSystemPathDataFiles=/var/lib/mysqlcluster/
DataMemory=16483M
IndexMemory=2686M
LockPagesInMainMemory=1

MaxNoOfConcurrentOperations=32768
MaxNoOfConcurrentTransactions=16384

StringMemory=25
MaxNoOfTables=10320
MaxNoOfOrderedIndexes=512
MaxNoOfUniqueHashIndexes=512
MaxNoOfAttributes=18000
MaxNoOfTriggers=71120
DiskCheckpointSpeedInRestart=100M
FragmentLogFileSize=256M
InitFragmentLogFiles=SPARSE
NoOfFragmentLogFiles=84
RedoBuffer=1000M
TransactionBufferMemory=8M

TimeBetweenLocalCheckpoints=20
TimeBetweenEpochsTimeout=32000
TimeBetweenGlobalCheckpoints=2000
TimeBetweenEpochs=100

### Heartbeating 
HeartbeatIntervalDbDb=15000
HeartbeatIntervalDbApi=15000

### Params for setting logging 
MemReportFrequency=30
BackupReportFrequency=10
LogLevelStartup=15
LogLevelShutdown=15
LogLevelCheckpoint=8
LogLevelNodeRestart=15

### Params for BACKUP 
BackupMaxWriteSize=1M
BackupDataBufferSize=24M
BackupLogBufferSize=16M
BackupMemory=40M

### Params for ODIRECT 
#Reports indicates that odirect=1 can cause io errors (os err code 5) on some systems. You must test.
#ODirect=1

### Watchdog 
TimeBetweenWatchdogCheckInitial=60000

### TransactionInactiveTimeout  - should be enabled in Production 
TransactionInactiveTimeout=60000
### New 7.1.10 redo logging parameters 
RedoOverCommitCounter=3
RedoOverCommitLimit=20
### REALTIME EXTENSIONS 
#RealTimeScheduler=1
### REALTIME EXTENSIONS FOR 6.3 ONLY
#SchedulerExecutionTimer=80
#SchedulerSpinTimer=40

### DISK DATA 
SharedGlobalMemory=384M
DiskPageBufferMemory=4024M
BatchSizePerLocalScan=1000M

[NDBD]
NodedId=2
HostName=127.0.0.1
#datadir=/usr/local/mysql/data
DataDir=/usr/local/mysql/data

[mysqld]
HostName=127.0.0.1
NodeId=3

[API]
NodeId=4
HostName=127.0.0.1

How to repeat:
dump data which is of 800MB with 400 table to mysql cluster
then it show this error one it dumps 102 tables no matter what parameters you change int config.ini file.
[17 Sep 2013 10:52] prasanth katta
i am facing problem while dumpig data to mysql cluster. when ever i dump the data (.sql which is of 800MB in size with 400 tables) in to mysql cluster i am getting the following error "No more table metadata records (increase MaxNoOfTables)' from NDBCLUSTER"
2013-09-17 09:15:41 [MgmtSrvr] INFO     -- Node 3: mysqld --server-id=0
2013-09-17 09:15:42 [MgmtSrvr] INFO     -- Node 2: Node 3 Connected
2013-09-17 09:15:42 [MgmtSrvr] INFO     -- Node 2: Node 3: API mysql-5.6.11 ndb-7.3.2
2013-09-17 09:27:32 [MgmtSrvr] INFO     -- Node 2: Data usage is 0%(8 32K pages of total 524288)
2013-09-17 09:27:32 [MgmtSrvr] INFO     -- Node 2: Index usage is 0%(12 8K pages of total 1310752)
2013-09-17 09:27:32 [MgmtSrvr] INFO     -- Node 2: Resource 0 min: 524407 max: 859521 curr: 331025
2013-09-17 09:27:32 [MgmtSrvr] INFO     -- Node 2: Resource 3 min: 851976 max: 851976 curr: 327696
2013-09-17 09:27:32 [MgmtSrvr] INFO     -- Node 2: Resource 5 min: 1152 max: 1152 curr: 1088
2013-09-17 09:27:32 [MgmtSrvr] INFO     -- Node 2: Resource 7 min: 2240 max: 2240 curr: 2240
2013-09-17 09:27:32 [MgmtSrvr] INFO     -- Node 2: Resource 9 min: 64 max: 0 curr: 1
[17 Sep 2013 11:44] Miguel Solorzano
Duplicate http://bugs.mysql.com/bug.php?id=70364 .
[17 Sep 2013 15:42] prasanth katta
config file of my setup

Attachment: config.ini (application/x-wine-extension-ini, text), 2.29 KiB.

[17 Sep 2013 15:43] prasanth katta
log file one

Attachment: ndb_1_cluster.log (text/x-log), 20.40 KiB.

[17 Sep 2013 15:43] prasanth katta
log file three

Attachment: ndb_2_out.log (text/x-log), 17.04 KiB.

[17 Sep 2013 15:44] prasanth katta
log file two

Attachment: ndb_1_out.log (text/x-log), 404 bytes.

[17 Sep 2013 15:56] prasanth katta
Hello Godofredo sorry for duplicates as i did not seen any edit option for the open ticket i had raised another one can you pls close that (Duplicate http://bugs.mysql.com/bug.php?id=70364)
[19 Sep 2013 6:42] Umesh Shastry
Hello Prasanth,

Thank you for the report.
I cannot repeat described behavior with 2000+ tables.
Could you tell us how you changed the value of MaxNoOfTables? 
Also, most importantly - For each attribute that has a BLOB data type an extra table is used to store most of the BLOB data. 
These tables also must be taken into account when defining the total number of tables.

I suspect that it is still using its default value (128) and after reaching 102 tables ( + extra for blob etc which are also counted towards this limit) it
is exhausting 128 default limit.

If you change this parameter in config.ini then you have to perform a rolling restart.

Also, see what ndb_config returns for MaxNoOfTables.. see, http://dev.mysql.com/doc/refman/5.6/en/mysql-cluster-programs-ndb-config.html

Also, Could you please attach the cluster logs? Preferably using the ndb_error_reporter utility:

  http://dev.mysql.com/doc/refman/5.5/en/mysql-cluster-programs-ndb-error-reporter.html
  
And, also provide mysqld error log where you are trying to import the dump file.

Thanks,
Umesh
[19 Sep 2013 10:13] prasanth katta
Hello Umesh,

Thank you for the reply.

i change MaxNoOfTables parameter in config.ini file (this is the value i am using for my setup MaxNoOfTables=20320).

i am attaching the required logs including mysqld logs.

please check the configinfo file attached,in the file it is showing the max value for MaxNoOfTables as 20320 but i think it is not taking that vale from config.ini file and setting it to default.

please look into it once.

here are the steps how i start my cluster just for you reference in case needed

1. ndb_mgmd -f /var/lib/mysql-cluster/config.ini (here it has to take all the valuse from this file but somehow it is not)

2. ndbd --initial 
 
3. finally mysql server ./mysqld_safe --user=xxx &

Thanks,
Prasanth Katta
[19 Sep 2013 10:14] prasanth katta
config info file

Attachment: configinfo (application/octet-stream, text), 54.57 KiB.

[19 Sep 2013 10:14] prasanth katta
mysqld logs

Attachment: mysqld.log (text/x-log), 30.06 KiB.

[19 Sep 2013 10:14] prasanth katta
ndb error report

Attachment: ndb_error_report_20130919093232.tar.bz2 (application/x-bzip2, text), 1.27 KiB.

[19 Sep 2013 10:51] Umesh Shastry
Okay, here is the problem, after making changes to MaxNoOfTables in config.ini file you started management node as:

1. ndb_mgmd -f /var/lib/mysql-cluster/config.ini (here it has to take all the valuse from this file but somehow it is not)

^^^^^ I see that you started ndb_mgmd without --initial or --reload, which means you didn't reload config file.. Configuration data is cached internally, rather than being read from the cluster global configuration file each time the management server is started. Using the --initial option overrides this behavior, by forcing the management server to delete any existing cache files, and then to re-read the configuration data from the cluster configuration file and to build a new cache. 

See, http://dev.mysql.com/doc/refman/5.6/en/mysql-cluster-programs-ndb-mgmd.html#option_ndb_mgm...

Also, could you pull config details? Don't pull from config.ini

ndb_config --type=ndbd --query=MaxNoOfTables --fields=':' --rows='\n'
[19 Sep 2013 11:54] prasanth katta
Hello Umesh,

Thanks a lot for you help here..now am able to see the updated global config file ..

here is the output of the query 
ndb_config --query=id,type --fields=':' --rows='\n'
20320

now i am able to update the other tables....

appreciate the help here..
[17 Dec 2017 18:46] Prakash Singh
Deal All,
i m getting the same error "No more table metadata records (increase MaxNoOfTables)' from NDBCLUSTER" but my max no tables parameter is at maximum 20 k plus also its refelecting in ndb_config -q MaxNoofTables still i m geeting same error why so i m using ndb 7.4/10 and mysql 5.6 version please help..guys