MySQL Bugs: #54268: MySQL Fails to start after normal shutdown with a table that is "full"

Bug #54268	MySQL Fails to start after normal shutdown with a table that is "full"
Submitted:	6 Jun 2010 18:33	Modified:	7 Jul 2010 13:07
Reporter:	Anders Karlsson	Email Updates:
Status:	No Feedback	Impact on me:	None
Category:	MySQL Cluster: Cluster (NDB) storage engine	Severity:	S3 (Non-critical)
Version:	mysql-5.1-telco-7.1	OS:	Windows
Assigned to:		CPU Architecture:	Any
Tags:	7.1.3

Description:
Trying to start MySQL Cluster ndbd after having shut it dopwn normally when a tablee is "full" generates errors, and ndbd refuses to start.

How to repeat:
Start MySQL Cluster on Windows and create a simple table (I used "CREATE TABLE t1 AS c1(int)". Insert a bunch of rows until you get this:
mysql> insert into t1 select * from t1 limit 5;
ERROR 1114 (HY000): The table 't1' is full
Actually, when this happens, you may still be able to insert more data, every third or so INSERT will succeed, those in between will have the same "table is full" errors, so this may be the root cause here.
Having this situation, shut down MySQL normally (ndb_mgm> SHUTDOWN;). Then restrt the mgmd and then the data nodes. The data nodes will fail to start with errors like this:
2010-06-06 20:25:08 [ndbd] INFO     -- Configuration fetched from 'localhost:118
6', generation: 2
2010-06-06 20:25:08 [ndbd] INFO     -- Ndb started
NDBMT: non-mt
2010-06-06 20:25:08 [ndbd] INFO     -- NDB Cluster -- DB node 3
2010-06-06 20:25:08 [ndbd] INFO     -- mysql-5.1.44 ndb-7.1.3 --
2010-06-06 20:25:08 [ndbd] INFO     -- Ndbd_mem_manager::init(1) min: 134Mb init
ial: 154Mb
Adding 155Mb to ZONE_LO (1,4935)
2010-06-06 20:25:08 [ndbd] INFO     -- Start initiated (mysql-5.1.44 ndb-7.1.3)
WOPool::init(61, 9)
RWPool::init(22, 13)
RWPool::init(42, 18)
RWPool::init(62, 13)
Using 1 fragments per node
RWPool::init(c2, 18)
RWPool::init(e2, 16)
WOPool::init(41, 8)
RWPool::init(82, 12)
RWPool::init(a2, 54)
WOPool::init(21, 8)
2010-06-06 20:25:08 [ndbd] INFO     -- Start phase 0 completed
2010-06-06 20:25:41 [ndbd] INFO     -- findNeighbours from: 2043 old (left: 6553
5 right: 65535) new (2 2)
2010-06-06 20:25:41 [ndbd] INFO     -- Start phase 1 completed
2010-06-06 20:25:42 [ndbd] INFO     -- Start phase 2 completed
2010-06-06 20:25:42 [ndbd] INFO     -- Start phase 3 completed
restartCreateObj(1) file: 1
restartCreateObj(2) file: 1
restartCreateObj(3) file: 1
restartCreateObj(4) file: 1
restartCreateObj(5) file: 1
restartCreateObj(6) file: 1
restartCreateObj(7) file: 1
RESTORE table: 2 1039 rows applied
RESTORE table: 2 1013 rows applied
RESTORE table: 3 2 rows applied
RESTORE table: 3 2 rows applied
RESTORE table: 4 1 rows applied
RESTORE table: 4 0 rows applied
RESTORE table: 5 0 rows applied
RESTORE table: 5 0 rows applied
RESTORE table: 6 0 rows applied
RESTORE table: 6 0 rows applied
RESTORE table: 7 752270 rows applied
2010-06-06 20:25:56 [ndbd] INFO     -- Error 625 (line: 1184) during restore of
 0/T7F1
2010-06-06 20:25:56 [ndbd] INFO     -- RESTORE (Line: 1209) 0x00000002
error=2352
2010-06-06 20:25:56 [ndbd] INFO     -- Error handler startup shutting down syste
m
2010-06-06 20:25:56 [ndbd] INFO     -- Error handler shutdown completed - exitin
g
2010-06-06 20:25:56 [ndbd] ALERT    -- Node 3: Forced node shutdown completed. O
ccured during startphase 4.

Please provide ndb_error_reporter output.

http://orgs.man.ac.uk/documentation/mysql/5.1/mysql-cluster-utilities-ndb-error-reporter.h...

Please see bug #54291. ndb_error_reporter is not available on Windows.

No feedback was provided for this bug for over a month, so it is
being suspended automatically. If you are able to provide the
information that was originally requested, please do so and change
the status of the bug back to "Open".

Again, output from ndb_error_reporter cannot be provided as this utility is not supported on Windows (which is bug #54291). Note that this is a crashing bug!

Tested again in 7.1.4b on Windows, with the same effect. This is a craching bug, and ndb_error_reporter still isn't available on Windows. I have observed the same issue on Linux though, again also in 7.1.4b and again reproducible in the same way. I will upload that ndb_error_reporter file.

ndb_error_reported file

Attachment: ndb_error_report_20100729210251.tar.bz2 (application/octet-stream, text), 34.08 KiB.