MySQL Bugs: #68703: Pointer too large error on data node startup

Bug #68703	Pointer too large error on data node startup
Submitted:	18 Mar 2013 13:43	Modified:	6 Jul 2016 14:26
Reporter:	Nenad Merdanovic	Email Updates:
Status:	No Feedback	Impact on me:	None
Category:	MySQL Cluster: Cluster (NDB) storage engine	Severity:	S2 (Serious)
Version:	mysql-5.5.29 ndb-7.2.10	OS:	Linux (Debian Squeeze)
Assigned to:	MySQL Verification Team	CPU Architecture:	Any
Tags:	pointer too large startup error

Description:
When (re)starting a NDB data node it crashes in startphase 5 with:

Node 2: Forced node shutdown completed. Occured during startphase 5. Caused by error 2306: 'Pointer too large(Internal error, programming error or missing error message, please report a bug). Temporary error, restart node'.

Error log:
Time: Monday 18 March 2013 - 08:09:13
Status: Temporary error, restart node
Message: Pointer too large (Internal error, programming error or missing error message, please report a bug)
Error: 2306
Error data: DblqhMain.cpp
Error object: DBLQH (Line: 22180) 0x00000002
Program: ndbd
Pid: 7832
Version: mysql-5.5.29 ndb-7.2.10
Trace: /home/mysql/data/ndb_2_trace.log.19 [t1..t1]

I am attaching config.ini and relevant traces. I can't run ndb_error_reporting without cluttering the output because of other bug I encountered and reported (ID: 68686). So if you need any more data, please let me know.

How to repeat:
Happens on random restarts, either from the ndb_mgm or by running ndbd in the shell.

Configuration file

Attachment: config.ini (application/octet-stream, text), 1.89 KiB.

Relevant trace file

Attachment: ndb_2_trace.log.19.tar.gz (application/x-gzip, text), 53.49 KiB.

This is a really stopper as any time the cluster falls apart it needs to be started with --initial. This means restoring from backup and losing data. Is there any more information I can provide so you can find the solution faster?

The same problem in ndb-7.3.5 happend the same.

The problem happens after an abnormal shutdown

Status: Temporary error, restart node
Message: Pointer too large (Internal error, programming error or missing error message, please report a bug)
Error: 2306
Error data: DbdihMain.cpp
Error object: DBDIH (Line: 17644) 0x00000002
Program: ndbd
Pid: 12730
Version: mysql-5.6.17 ndb-7.3.5
Trace: /app/mysql-cluster/ndb_4_trace.log.18 [t1..t1]
***EOM***

////////////////

2014-09-04 05:37:37 [MgmtSrvr] INFO     -- Node 1: Node 4: API mysql-5.6.17 ndb-7.3.5
2014-09-04 05:37:37 [MgmtSrvr] INFO     -- Node 1: Node 6: API mysql-5.6.17 ndb-7.3.5
2014-09-04 05:37:37 [MgmtSrvr] INFO     -- Node 1: Start phase 1 completed
2014-09-04 05:37:37 [MgmtSrvr] INFO     -- Node 4: Start phase 1 completed
2014-09-04 05:37:37 [MgmtSrvr] INFO     -- Node 1: Start phase 2 completed (system restart)
2014-09-04 05:37:37 [MgmtSrvr] INFO     -- Node 6: Start phase 2 completed (system restart)
2014-09-04 05:37:37 [MgmtSrvr] INFO     -- Node 4: Start phase 2 completed (system restart)
2014-09-04 05:37:37 [MgmtSrvr] INFO     -- Node 1: Start phase 3 completed (system restart)
2014-09-04 05:37:37 [MgmtSrvr] INFO     -- Node 4: Start phase 3 completed (system restart)
2014-09-04 05:37:37 [MgmtSrvr] INFO     -- Node 6: Start phase 3 completed (system restart)
2014-09-04 05:37:37 [MgmtSrvr] INFO     -- Node 4: Restarting cluster to GCI: 5012469
2014-09-04 05:37:37 [MgmtSrvr] INFO     -- Node 4: Starting to restore schema
2014-09-04 05:37:42 [MgmtSrvr] ALERT    -- Node 4: Forced node shutdown completed. Occured during startphase 4. Caused by error 2306: 'Pointer too large(Internal error, programming error or missing error message, please report a bug). Temporary error, restart node'.
2014-09-04 05:37:42 [MgmtSrvr] ALERT    -- Node 101: Node 4 Disconnected
2014-09-04 05:37:43 [MgmtSrvr] ALERT    -- Node 2: Forced node shutdown completed. Occured during startphase 2. Caused by error 2308: 'Another node failed during system restart, please investigate error(s) on other node(s)(Restart error). Temporary error, restart node'.
2014-09-04 05:37:43 [MgmtSrvr] ALERT    -- Node 6: Forced node shutdown completed. Occured during startphase 4. Caused by error 2308: 'Another node failed during system restart, please investigate error(s) on other node(s)(Restart error). Temporary error, restart node'.
2014-09-04 05:37:43 [MgmtSrvr] ALERT    -- Node 1: Forced node shutdown completed. Occured during startphase 4. Caused by error 2308: 'Another node failed during system restart, please investigate error(s) on other node(s)(Restart error). Temporary error, restart node'.

tracefiles

Attachment: mysql-bug-data-20140409-68703.tar.gz (application/x-gzip, text), 131.23 KiB.

I can reproduce the error after a hot shut down of 3 machines in the cluster.

When I´ve try to restart three of the nodes with --initial, I get the error again.
Let me know if you need more information.
Thanks

Hi,

we need a full ndb_error_reported output, not just the trace files

kind regards
Bogdan Kecman

No feedback was provided for this bug for over a month, so it is
being suspended automatically. If you are able to provide the
information that was originally requested, please do so and change
the status of the bug back to "Open".

I'm having this issue. I receive the following error message and the node goes down.
Time: Friday 12 May 2017 - 17:15:24
Status: Temporary error, restart node
Message: Pointer too large (Internal error, programming error or missing error message, please report a bug)
Error: 2306
Error data: DblqhMain.cpp
Error object: DBLQH (Line: 17777) 0x00000002
Program: ndbd
Pid: 4157
Version: mysql-5.7.18 ndb-7.5.6
Trace file name: ndb_3_trace.log.2
Trace file path: /data/mysql/data/ndb_3_trace.log.2 [t1..t1]
***EOM***

it's on Centos 7 (3.10.0-123.el7.x86_64)

i can consistently reproduce this. if you want logs, let me know. thanks.