Bug #20933 Forced node shutdown completed. Initiated by signal 11. Caused by error 6000
Submitted: 10 Jul 2006 2:42 Modified: 12 Jul 2006 10:13
Reporter: Willy Ang Email Updates:
Status: Closed Impact on me:
None 
Category:MySQL Server Severity:S3 (Non-critical)
Version:5.0.22 OS:Linux (CentOS 4.3)
Assigned to: CPU Architecture:Any

[10 Jul 2006 2:42] Willy Ang
Description:
Hi,

I've setup a MySQL 5.0.22 cluster on CentOS 4.3 . In my cluster, I have:

- 2 SQL nodes
- 2 NDB nodes
- 1 MGM node

Only the SQL nodes are on publicly accessible IP addresses. The NDB nodes and MGM nodes are on private IP addresses. The SQL nodes can access the NDB nodes and MGM nodes.

The cluster starts up fine. I'm able to run some queries on the tables through the mysql console. However when I execute a script which runs several SQL commands, one of the NDB nodes will die with the following error:

2006-07-03 15:32:49 [MgmSrvr] ALERT -- Node 2: Forced node shutdown completed. Initiated by signal 11. Caused by error 6000: 'Error OS signal received(Internal error, programming error or missing error message, please report a bug). Temporary error, restart node'.

The following appears in the ndb_2_error.log:

-----------------------------------------------------------------
Time: Monday 3 July 2006 - 15:32:48
Status: Temporary error, restart node
Message: Error OS signal received (Internal error, programming error or missing error message, please report a bug)
Error: 6000
Error data: Signal 11 received; Segmentation fault
Error object: main.cpp
Program: ndbd
Pid: 13899
Trace: /var/lib/mysql-cluster/ndb_2_trace.log.16
Version: Version 5.0.22
-----------------------------------------------------------------

The remaining NDB node becomes the master. When I re-run the same script, this second NDB node also crashes and the entire NDB is down. However I'm able to restart the NDB on each of the NDB nodes with the "ndbd" command.

My config.ini file is:

------------------------------------------------
[NDBD DEFAULT]
NoOfReplicas=2
DataDir=/var/lib/mysql-cluster
MaxNoOfOrderedIndexes=512
MaxNoOfUniqueHashIndexes=512
DataMemory=160M
IndexMemory=72M

# Management Node
[NDB_MGMD]
HostName=192.168.2.2
DataDir=/var/lib/mysql-cluster

# Storage Nodes
# One entry for each node
[NDBD]
HostName=192.168.2.5
[NDBD]
HostName=192.168.2.6

# SQL Nodes
# One entry for each node
[MYSQLD]
HostName=123.123.1.1 # actual external IP address changed
[MYSQLD]
HostName=123.123.1.2 # actual external IP address changed
------------------------------------------------ 

How to repeat:
By running the script which executes several SQL queries, the NDB node will crash again.

Suggested fix:
I've to manually restart ndbd process each time the NDB node crashes.
[10 Jul 2006 9:36] Hartmut Holzgraefe
Hi, we'll need the script you're running to create a reproduceable test case,
can you upload the script and all other files needed to reproduce the problem
(CREATE statements etc.) to the "Files" section of this bug report or to our
FTP server?