Bug #29383 ndbd not connect with mgmd
Submitted: 27 Jun 2007 11:45 Modified: 10 Aug 2007 13:43
Reporter: Sany Das Email Updates:
Status: No Feedback Impact on me:
None 
Category:MySQL Cluster: Cluster (NDB) storage engine Severity:S1 (Critical)
Version:5.1.16 OS:Linux (can't connect ndbd nodes with mgmd node)
Assigned to: CPU Architecture:Any

[27 Jun 2007 11:45] Sany Das
Description:
I have 2 datanodes, 2 API nodes and 1 management nodes 
i restart the mgmd node " ndb_mgmd -f /var/lib/mysql-cluster/config.ini"
then I restart ndbd nodes

but when I checked the status on management node with 'SHOW' command it can't coneect the data nodes and showing the following errors:

ndb_mgm> Node 3: Forced node shutdown completed. Occured during startphase 4. Caused by error 2807: 'File has already been opened(Internal error, programming error or missing error message, please report a bug). Temporary error, restart node'. - Unknown error code: Unknown result: Unknown error code

Please suggest

How to repeat:
100% reproducible
[28 Jun 2007 8:20] Sveta Smirnova
Thank you for the report.

I can not repeat described behaviour in my environment.

Additionally version 5.1.16 is quite old. Please upgrade to current version 5.1.19 and if you can repeat described behaviour upload cluster configuration and log files.
[28 Jun 2007 8:21] Sveta Smirnova
Also please provide sequence of commands you use to shutdown and start management and data nodes.
[29 Jun 2007 10:35] Sany Das
The configuration files are

" my.cnf  "  in data nodes
=========================
[client]				 
port	= 3306				 
socket	= /var/lib/mysql/mysql.sock	 
[mysqld]				 
port	= 3306				 
socket	= /var/lib/mysql/mysql.sock	 
datadir = /var/lib/mysql		 
back-log=75
skip-innodb
max_connections=500
skip-locking				 
key_buffer = 384M			
myisam_sort_buffer_size=64m
join_buffer_size=1M
max_allowed_packet = 64M			 
table_cache = 1800
thread_cache_size=384
wait_timeout=7200
connect_timeout=10
tmp_table_size=64M
max_heap_table_size=64M
max_connect_errors=1000
read_rnd_buffer_size=524288
bulk_insert_buffer_size=8M
sort_buffer_size = 2M			 
net_buffer_length = 8K			 
read_buffer_size = 1M			 
log-bin=mysql-bin			 
query_cache_limit=4M
query_cache_size=128M
query_cache_type=1
query_prealloc_size=65536
querry_alloc_block-size=131072

[mysqld_safe]
nice=-5
open_files_limit=8192

[mysqldump]				 
quick					 
max_allowed_packet = 16M		 
[mysql]					 
no-auto-rehash				 

[myisamchk]				 
key_buffer = 64M			 
sort_buffer_size = 64M			 
read_buffer = 16M 			 
write_buffer = 16M			 
  			 
[mysqlhotcopy]				 
interactive-timeout  			 
[mysqld]				 
ndbcluster 				 
ndb-connectstring=192.168.3.97 
[mysql_cluster] 			 
ndb-connectstring=192.168.3.97  

and "config.ini" in mgmd node is
================================

[NDBD DEFAULT]				
NoOfReplicas=2				
DataMemory=600M				
IndexMemory=200M	
RedoBuffer=300M

		
						
# TCP/IP options:				
[TCP DEFAULT]				
portnumber=2202				
						
# Management process options:		
[NDB_MGMD] 				
						
hostname=192.168.3.97  	# Hostname or IP address of MGM node		
datadir=/var/lib/mysql-cluster	# Directory for MGM node logfiles		

						
						
							
# Options for data node 1 :			
[NDBD]						
		# (one [NDBD] section per data node)	

RedoBuffer=500M
DataMemory=256M
IndexMemory=256M							
hostname=192.168.3.99	# Hostname or IPs 	
datadir=/var/lib/mysql	# path for data node's datafiles 
MaxNoOfAttributes=15000	
TimeBetweenLocalCheckPoints=6
TimeBetweenGlobalCheckPoints=10			
MaxNoOfTables=1000				
MaxNoOfOrderedIndexes=10000			
MaxNoOfUniqueHashIndexes=10000
MaxNoOfConcurrentTransactions=5000			
MaxNoOfConcurrentOperations=250000            	
TransactionBufferMemory=10M
NoOfFragmentLogFiles=300
TimeBetweenWatchDogCheck=600000
MaxNoOfLocalOperations=275000			
							
							
# Options for data node 2 :			
[NDBD]						
		# (one [NDBD] section per data node)	

RedoBuffer=500M
DataMemory=256M	
IndexMemory=256M					
hostname=192.168.3.100	# Hostname or IPs 	
datadir=/var/lib/mysql	# path for data node's datafiles 
MaxNoOfAttributes=15000				
TimeBetweenLocalCheckPoints=6
TimeBetweenGlobalCheckPoints=10
MaxNoOfTables=1000				
MaxNoOfOrderedIndexes=10000			
MaxNoOfUniqueHashIndexes=10000	
MaxNoOfConcurrentTransactions=5000		
MaxNoOfConcurrentOperations=250000
TransactionBufferMemory=10M
NoOfFragmentLogFiles=300
TimeBetweenWatchDogCheck=600000            	
MaxNoOfLocalOperations=275000			
							
	# SQL node options:				
[mysqld]					 
hostname=192.168.3.96		# SQL Node IP
[mysqld]
hostname=192.168.3.95		# SQL/Offline IP	 	 

How to repeat:
set cluster with atleast two data nodes

and try to insert contineously into ndbd tables

observe the result after some time
[10 Jul 2007 13:43] Hartmut Holzgraefe
To fully analyse the error we need the full cluster logs and trace files from the management node and the failing data node(s). You can use the ndb_error_reporter tool for this which collects and archives all necessary files using ssh/scp, see http://dev.mysql.com/doc/refman/5.1/en/mysql-cluster-utilities-ndb-error-reporter.html

Please also provide sequence of commands you use to shutdown and start management and data nodes (as already requested).
[10 Aug 2007 23:00] Bugs System
No feedback was provided for this bug for over a month, so it is
being suspended automatically. If you are able to provide the
information that was originally requested, please do so and change
the status of the bug back to "Open".