MySQL Bugs: #59592: MySQL Cluster 7.1.9 whole cluster crash Error: 2306 Pointer too large

Bug #59592	MySQL Cluster 7.1.9 whole cluster crash Error: 2306 Pointer too large
Submitted:	18 Jan 2011 20:13	Modified:	1 Mar 2011 9:24
Reporter:	Carl Krumins	Email Updates:
Status:	No Feedback	Impact on me:	None
Category:	MySQL Cluster: Cluster (NDB) storage engine	Severity:	S2 (Serious)
Version:	mysql-5.1.51 ndb-7.1.9	OS:	Linux (Ubuntu 2.6.32-25-server)
Assigned to:		CPU Architecture:	Any
Tags:	mysql cluster 2306

Description:

Time: Wednesday 19 January 2011 - 06:24:37
Status: Temporary error, restart node
Message: Pointer too large (Internal error, programming error or missing error message, please report a bug)
Error: 2306
Error data: dbdih/DbdihMain.cpp
Error object: DBDIH (Line: 17161) 0x00000002
Program: ndbmtd
Pid: 12523 thr: 0
Version: mysql-5.1.51 ndb-7.1.9
Trace: /data/mysqlcluster//ndb_3_trace.log.4 /data/mysqlcluster//ndb_3_trace.log.4_t1 /data/mysqlcluster//ndb_3_trace.log.4_t2 /data/mysqlcluster//ndb_3_

restarted single client mysqld node on latest cluster configured from severalnines.com caused whole cluster to crash.

2011-01-19 06:20:30 [MgmtSrvr] WARNING  -- Node 5: Failure handling of node 87 has not completed in 1 min - state = 7
2011-01-19 06:21:30 [MgmtSrvr] WARNING  -- Node 5: Failure handling of node 87 has not completed in 2 min - state = 7
..
2011-01-19 06:24:31 [MgmtSrvr] WARNING  -- Node 5: Failure handling of node 87 has not completed in 5 min - state = 7

2011-01-19 06:25:47 [MgmtSrvr] ALERT    -- Node 5: Forced node shutdown completed. Caused by error 2306: 'Pointer too large(Internal error, programming error or missing error message, please report a bug). Temporary error, restart node'.
2011-01-19 06:26:28 [MgmtSrvr] ALERT    -- Node 3: Forced node shutdown completed. Caused by error 2306: 'Pointer too large(Internal error, programming error or missing error message, please report a bug). Temporary error, restart node'.
and whole cluster bought down.

[TCP DEFAULT]
SendBufferMemory=8M
ReceiveBufferMemory=8M

[NDB_MGMD DEFAULT]
PortNumber=1186
Datadir=/data/mysqlcluster/

[NDB_MGMD]
NodeId=1
Hostname=192.168.5.5
LogDestination=FILE:filename=ndb_1_cluster.log,maxsize=10000000,maxfiles=6
ArbitrationRank=1

[NDBD DEFAULT]
NoOfReplicas=2
LongMessageBuffer=16M
#DataMemory=51200M
DataMemory=46080M
IndexMemory=6400M
LockPagesInMainMemory=1
MaxNoOfConcurrentScans=500
MaxNoOfLocalScans=1000
MaxNoOfConcurrentOperations=40000000
MaxNoOfConcurrentIndexOperations=1000000
MaxNoOfConcurrentTransactions=500000
MaxNoOfLocalOperations=50000000
MaxNoOfSubscribers=90112

StringMemory=35
#MaxNoOfTables=8096
MaxNoOfTables=7224
MaxNoOfOrderedIndexes=16384
MaxNoOfUniqueHashIndexes=1024
MaxNoOfAttributes=259072
MaxNoOfTriggers=8024

CompressedBACKUP=1
CompressedLCP=1
DiskIOThreadPool=64
UndoIndexBuffer=8M
UndoDataBuffer=64M
RedoBuffer=64M

### Params for REDO LOG
FragmentLogFileSize=256M
InitFragmentLogFiles=SPARSE
NoOfFragmentLogFiles=624

TimeBetweenGlobalCheckpoints=1000
TimeBetweenEpochs=100

TimeBetweenEpochsTimeout=32000

### Params for LCP
DiskCheckpointSpeedInRestart=200M
DiskCheckpointSpeed=20M
TimeBetweenLocalCheckpoints=20

### Heartbeating
HeartbeatIntervalDbDb=15000
HeartbeatIntervalDbApi=15000

### Params for setting logging
MemReportFrequency=30
BackupReportFrequency=10
LogLevelStartup=15
LogLevelShutdown=15
LogLevelCheckpoint=8
LogLevelNodeRestart=15

### Params for BACKUP
BackupMaxWriteSize=1M
BackupDataBufferSize=16M
BackupLogBufferSize=4M
BackupMemory=20M

### Params for ODIRECT
#Reports indicates that odirect=1 can cause io errors (os err code 5) on some systems. You must test.
ODirect=1

### Watchdog
TimeBetweenWatchdogCheckInitial=60000

### TransactionInactiveTimeout  - should be enabled in Production
TransactionInactiveTimeout=60000
### REALTIME EXTENSIONS
RealTimeScheduler=1
### REALTIME EXTENSIONS FOR 6.3 ONLY
SchedulerExecutionTimer=100
SchedulerSpinTimer=40

### DISK DATA
SharedGlobalMemory=512M
DiskPageBufferMemory=5120M
BatchSizePerLocalScan=512

[NDBD]
NodeId=2
Hostname=192.168.5.10
MaxNoOfExecutionThreads=8
Datadir=/data/mysqlcluster/
FileSystemPathDD=/data/mysqlcluster/
FileSystemPathUndoFiles=/data/mysqlcluster/Undo/
FileSystemPathDataFiles=/data/mysqlcluster/Data/

[NDBD]
NodeId=3
Hostname=192.168.5.20
MaxNoOfExecutionThreads=8
Datadir=/data/mysqlcluster/
FileSystemPathDD=/data/mysqlcluster/
FileSystemPathUndoFiles=/data/mysqlcluster/Undo/
FileSystemPathDataFiles=/data/mysqlcluster/Data/

[NDBD]
NodeId=4
Hostname=192.168.5.11
MaxNoOfExecutionThreads=8
Datadir=/data4/mysqlcluster/
FileSystemPathDD=/data4/mysqlcluster/
FileSystemPathUndoFiles=/data4/mysqlcluster/Undo/
FileSystemPathDataFiles=/data4/mysqlcluster/Data/

[NDBD]
NodeId=5
Hostname=192.168.5.21
MaxNoOfExecutionThreads=8
Datadir=/data5/mysqlcluster/
FileSystemPathDD=/data5/mysqlcluster/
FileSystemPathUndoFiles=/data5/mysqlcluster/Undo/
FileSystemPathDataFiles=/data5/mysqlcluster/Data/

How to repeat:
not sure, crashed on production system while mysqladmin -p shutdown on a single mysqld node

Please attach all logs using ndb_error_reporter
If all cluster went down more nodes must have failed.

No feedback was provided for this bug for over a month, so it is
being suspended automatically. If you are able to provide the
information that was originally requested, please do so and change
the status of the bug back to "Open".