MySQL Bugs: #19378: Job Buffer Full error appears for some range of MaxNoOfTables

Bug #19378	Job Buffer Full error appears for some range of MaxNoOfTables
Submitted:	26 Apr 2006 16:54	Modified:	23 Mar 2007 1:31
Reporter:	Serge Kozlov	Email Updates:
Status:	Closed	Impact on me:	None
Category:	MySQL Cluster: Cluster (NDB) storage engine	Severity:	S2 (Serious)
Version:	4.1 ->	OS:	Linux (Linux FC4)
Assigned to:	Guangbao Ni	CPU Architecture:	Any

Description:
Seems this bug like #19352
For some range of MaxNoOfTables (probably the range depends by RAM) a data nodes couldn't start and return the error:

Current byte-offset of file-pointer is: 568

Time: Wednesday 26 April 2006 - 18:39:05
Status: Temporary error, restart node
Message: Job buffer congestion (Internal error, programming error or missing err
or message, please report a bug)
Error: 2334
Error data: Job Buffer Full
Error object: APZJobBuffer.C
Program: /home/ndbdev/skozlov/builds/libexec/ndbd
Pid: 27967
Trace: /space/run/ndb_3_trace.log.1
Version: Version 5.1.10 (beta)
***EOM***

How to repeat:
1. Set 'middle' value for MaxNoOfTables. E.g. MaxNoOfTables = 500K
2. Start a cluster
3. Look at ndb_X_error.log

Suggested fix:
As for #19352 probably cluster should return "memory allocation" error ...

after patched the patch of bug#19352, it can give the expected error message "Memory allocation failure"

No feedback was provided for this bug for over a month, so it is
being suspended automatically. If you are able to provide the
information that was originally requested, please do so and change
the status of the bug back to "Open".

Checked in 5.1-bk with small changes against original description: were added following options into config.ini:
MaxNoOfTables: 400K
DataMemory: 1G

And getting following same error:

Time: Thursday 7 December 2006 - 22:11:03
Status: Temporary error, restart node
Message: Job buffer congestion (Internal error, programming error or missing err
or message, please report a bug)
Error: 2334
Error data: Job Buffer Full
Error object: APZJobBuffer.C
Program: ./builds/libexec/ndbd
Pid: 14343
Trace: /space/run/ndb_3_trace.log.1
Version: Version 5.1.15 (beta)
***EOM***

I alway get the following error messages even though i have increased these parameters.
DataMemory= 4G
IndexMemory= 1G
MaxNoOfTables= 50M

My test machine is 8G memory.

Current byte-offset of file-pointer is: 1067

Time: Friday 15 December 2006 - 10:50:06
Status: Permanent error, external action needed
Message: Memory allocation failure, please decrease some configuration parameters (Configuration error)
Error: 2327
Error data: DBTUP could not allocate memory for Fragrecord
Error object: Requested: 288x10486916 = 3020231808 bytes
Program: ./libexec/ndbd
Pid: 9506
Trace: /home/mysql-5.1/mysql-cluster/node2/ndb_2_trace.log.1
Version: Version 5.1.14 (beta)
***EOM***

Do not use maximal values for NoOfTables. The bug can be repeated for limited range of values. E.g. for DataMemory=1G need to investigate range 100K ...500K with step 50K: for NoOfTables=100K the cluster started successfully, for NoOfTables=500K node crashes with error "DBTUP could not allocate memory for Fragrecord".

I found when DataMemory= 1G and MaxNoOfTables=115K get error and exit abnormally, but I still can't get the messages just as you descripted.
  I still found that the newest version about mysql-5.1 main tree that i cloned it  is  5.1.15.

hi serge, I still can't reproduce it locally, could you upload your error file, trace file and config file?
Thank you very much!

A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/19380

ChangeSet@1.2420, 2007-02-06 20:00:00+08:00, gni@dev3-221.dev.cn.tlan +2 -0
  BUG#19378 Job Buffer Full error appears for too large MaxNoOfTables

A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/19678

ChangeSet@1.2260, 2007-02-12 16:35:32+08:00, gni@dev3-221.dev.cn.tlan +1 -0
  BUG#19378 Job Buffer Full error appears for too large MaxNoOfTables

push into mysql-5.1-new-ndb-bj

push into mysql-5.0-ndb-bj and mysql-5.1-new-ndb-bj

Pushed to main 5.0 and 5.1. Please set to Documenting if you don't intend to fix it in 4.1.

That's 5.0.40 and 5.1.17

> Please set to Documenting if you don't intend to fix it in 4.1.

Let's not make people guess. In future, please *specify*: "WONTFIX in X.X".

Thanks!

Thank you for your bug report. This issue has been committed to our source repository of that product and will be incorporated into the next release.

If necessary, you can access the source repository and build the latest available version, including the bug fix. More information about accessing the source trees is available at

    http://dev.mysql.com/doc/en/installing-source.html

Documented bugfix in 5.0.40 and 5.1.17 changelogs.