Bug #65703 Node crash after c_gcp_list.seize() failed
Submitted: 21 Jun 2012 23:04 Modified: 30 Jun 2016 14:52
Reporter: Morten Tryfoss Email Updates:
Status: Can't repeat Impact on me:
None 
Category:MySQL Cluster: Cluster (NDB) storage engine Severity:S2 (Serious)
Version:7.1.10 OS:Linux (CentOS 5.7 2.6.18-274.18.1.el5)
Assigned to: MySQL Verification Team CPU Architecture:Any

[21 Jun 2012 23:04] Morten Tryfoss 
Description:
We've had a full crash on our system.

After looking at the logs we got a large numbers of these lines prior to the crash:
2012-06-21 09:50:27 [ndbd] ERROR    -- c_gcp_list.seize() failed: gci: 250364759059202058 nodes: 00000000000000000000000000000000000000000000000000000feffffffff0
2012-06-21 09:50:27 [ndbd] WARNING  -- ACK wo/ gcp record (gci: 58292588/10) ref: 0fa2001b from: 0fa2001b
2012-06-21 09:50:27 [ndbd] WARNING  -- ACK wo/ gcp record (gci: 58292588/10) ref: 0fa20021 from: 0fa20021
2012-06-21 09:50:27 [ndbd] WARNING  -- ACK wo/ gcp record (gci: 58292588/10) ref: 0fa2000f from: 0fa2000f
2012-06-21 09:50:27 [ndbd] WARNING  -- ACK wo/ gcp record (gci: 58292588/10) ref: 0fa2000a from: 0fa2000a
2012-06-21 09:50:27 [ndbd] WARNING  -- ACK wo/ gcp record (gci: 58292588/10) ref: 0fa20019 from: 0fa20019
2012-06-21 09:50:27 [ndbd] WARNING  -- ACK wo/ gcp record (gci: 58292588/10) ref: 0fa20028 from: 0fa20028
2012-06-21 09:50:27 [ndbd] WARNING  -- ACK wo/ gcp record (gci: 58292588/10) ref: 0fa2001e from: 0fa2001e
2012-06-21 09:50:27 [ndbd] WARNING  -- ACK wo/ gcp record (gci: 58292588/10) ref: 0fa2002a from: 0fa2002a
2012-06-21 09:50:27 [ndbd] WARNING  -- ACK wo/ gcp record (gci: 58292588/10) ref: 0fa2002b from: 0fa2002b
2012-06-21 09:50:27 [ndbd] WARNING  -- ACK wo/ gcp record (gci: 58292588/10) ref: 0fa2000e from: 0fa2000e
2012-06-21 09:50:27 [ndbd] WARNING  -- ACK wo/ gcp record (gci: 58292588/10) ref: 0fa20012 from: 0fa20012
2012-06-21 09:50:27 [ndbd] WARNING  -- ACK wo/ gcp record (gci: 58292588/10) ref: 0fa20006 from: 0fa20006
2012-06-21 09:50:27 [ndbd] WARNING  -- ACK wo/ gcp record (gci: 58292588/10) ref: 0fa2001d from: 0fa2001d
2012-06-21 09:50:27 [ndbd] WARNING  -- ACK wo/ gcp record (gci: 58292588/10) ref: 0fa20023 from: 0fa20023
2012-06-21 09:50:27 [ndbd] WARNING  -- ACK wo/ gcp record (gci: 58292588/10) ref: 0fa20027 from: 0fa20027
2012-06-21 09:50:27 [ndbd] WARNING  -- ACK wo/ gcp record (gci: 58292588/10) ref: 0fa20013 from: 0fa20013
2012-06-21 09:50:27 [ndbd] WARNING  -- ACK wo/ gcp record (gci: 58292588/10) ref: 0fa20017 from: 0fa20017
2012-06-21 09:50:27 [ndbd] WARNING  -- ACK wo/ gcp record (gci: 58292588/10) ref: 0fa20004 from: 0fa20004
2012-06-21 09:50:27 [ndbd] WARNING  -- ACK wo/ gcp record (gci: 58292588/10) ref: 0fa20018 from: 0fa20018
2012-06-21 09:50:27 [ndbd] WARNING  -- ACK wo/ gcp record (gci: 58292588/10) ref: 0fa20020 from: 0fa20020
2012-06-21 09:50:27 [ndbd] WARNING  -- ACK wo/ gcp record (gci: 58292588/10) ref: 0fa20008 from: 0fa20008
2012-06-21 09:50:27 [ndbd] WARNING  -- ACK wo/ gcp record (gci: 58292588/10) ref: 0fa20007 from: 0fa20007
2012-06-21 09:50:27 [ndbd] WARNING  -- ACK wo/ gcp record (gci: 58292588/10) ref: 0fa20010 from: 0fa20010
2012-06-21 09:50:27 [ndbd] WARNING  -- ACK wo/ gcp record (gci: 58292588/10) ref: 0fa20011 from: 0fa20011
2012-06-21 09:50:27 [ndbd] WARNING  -- ACK wo/ gcp record (gci: 58292588/10) ref: 0fa2001a from: 0fa2001a
2012-06-21 09:50:27 [ndbd] WARNING  -- ACK wo/ gcp record (gci: 58292588/10) ref: 0fa2000c from: 0fa2000c
2012-06-21 09:50:27 [ndbd] WARNING  -- ACK wo/ gcp record (gci: 58292588/10) ref: 0fa20026 from: 0fa20026
2012-06-21 09:50:27 [ndbd] WARNING  -- ACK wo/ gcp record (gci: 58292588/10) ref: 0fa2001f from: 0fa2001f
2012-06-21 09:50:27 [ndbd] WARNING  -- ACK wo/ gcp record (gci: 58292588/10) ref: 0fa2000d from: 0fa2000d
2012-06-21 09:50:27 [ndbd] WARNING  -- ACK wo/ gcp record (gci: 58292588/10) ref: 0fa20016 from: 0fa20016
2012-06-21 09:50:27 [ndbd] WARNING  -- ACK wo/ gcp record (gci: 58292588/10) ref: 0fa20025 from: 0fa20025
2012-06-21 09:50:27 [ndbd] WARNING  -- ACK wo/ gcp record (gci: 58292588/10) ref: 0fa20014 from: 0fa20014
2012-06-21 09:50:27 [ndbd] WARNING  -- ACK wo/ gcp record (gci: 58292588/10) ref: 0fa2000b from: 0fa2000b
2012-06-21 09:50:27 [ndbd] WARNING  -- ACK wo/ gcp record (gci: 58292588/10) ref: 0fa2001c from: 0fa2001c
2012-06-21 09:50:27 [ndbd] WARNING  -- ACK wo/ gcp record (gci: 58292588/10) ref: 0fa20009 from: 0fa20009
2012-06-21 09:50:27 [ndbd] WARNING  -- ACK wo/ gcp record (gci: 58292588/10) ref: 0fa20015 from: 0fa20015
2012-06-21 09:50:27 [ndbd] WARNING  -- ACK wo/ gcp record (gci: 58292588/10) ref: 0fa20005 from: 0fa20005
2012-06-21 09:50:27 [ndbd] WARNING  -- ACK wo/ gcp record (gci: 58292588/10) ref: 0fa20022 from: 0fa20022
2012-06-21 09:50:27 [ndbd] WARNING  -- ACK wo/ gcp record (gci: 58292588/10) ref: 0fa20029 from: 0fa20029
2012-06-21 09:50:27 [ndbd] INFO     -- SimulatedBlock.cpp
2012-06-21 09:50:27 [ndbd] INFO     -- DBTC (Line: 1374) 0x00000006
2012-06-21 09:50:27 [ndbd] INFO     -- Error handler shutting down system
2012-06-21 09:50:27 [ndbd] INFO     -- Error handler shutdown completed - exiting
2012-06-21 09:50:27 [ndbd] ALERT    -- Node 2: Forced node shutdown completed. Caused by error 2341: 'Internal program error (failed ndbrequire)(Internal error, programming error or missing error message, please report a bug). Temporary error, restart node'.

It seems like it has been doing this for quite some time.
I can't find a bug related to it, so I'm doing as told - report it.

Configuration:
[ndbd default]
NoOfReplicas= 2
DataDir= /var/lib/mysql-cluster
DataMemory=4000M
IndexMemory=500M
MaxNoOfConcurrentOperations=1000000
MaxNoOfConcurrentTransactions = 32768
MaxNoOfOrderedIndexes=2048
MaxNoOfUniqueHashIndexes=1024
MaxNoOfAttributes=5000
MaxNoOfTables=256
TransactionDeadlockDetectionTimeout=10000
DiskCheckpointSpeed=30M
Diskcheckpointspeedinrestart = 200M
LockPagesInMainMemory=1
RedoBuffer=64M
NoOfFragmentLogFiles=22
InitFragmentLogFiles=SPARSE
FragmentLogFileSize=256M

SchedulerSpinTimer=400
SchedulerExecutionTimer=100
RealTimeScheduler=1
ODirect=1

TimeBetweenEpochsTimeout=16000
MaxNoOfExecutionThreads=4
MaxNoOfConcurrentScans=500

[tcp default]
SendBufferMemory=8M
ReceiveBufferMemory=8M

I'll upload the trace, but the logfile is way to big.

How to repeat:
I can't repeat it.
[21 Jun 2012 23:05] Morten Tryfoss 
trace

Attachment: trace.tar.gz (application/x-gzip-compressed, text), 266.27 KiB.
[30 Jun 2016 14:52] MySQL Verification Team 
can't reproduce (not even reported)
not enough data for trace analysis
4 years passed