MySQL Bugs: #29386: Cluster failure on contineous insertion

Bug #29386	Cluster failure on contineous insertion
Submitted:	27 Jun 2007 12:42	Modified:	29 Feb 2008 11:31
Reporter:	Sany Das	Email Updates:
Status:	No Feedback	Impact on me:	None
Category:	MySQL Cluster: Cluster (NDB) storage engine	Severity:	S3 (Non-critical)
Version:	5.1.16	OS:	Linux (Cluster failure on contineous insertion)
Assigned to:		CPU Architecture:	Any

Description:
We have 2 data nodes 1 mangement node and 2 api nodes

when we try to insert values (through a contineous loop) into table that is in ndb engine then each time the cluster fails after approx 1:30 hr showing error of different kinds as mentioned below

Error 1297: Got temporary error 'REDO' log overloaded.
Error 1205: Lock wait timeout exceeded by restarting transaction
Error 1297: Got temporary error 1204 'Temporary failure distribution changed' from  
NDBDCLUSTER
Error 1296: Got error 4009 'Cluster Failure' from NDBDCLUSTER

We tried to change some parameters in config.ini file in mgmd node as mentioned below but ended with same problem.

TimeBetweenLocalCheckPoints=6
TimeBetweenGlobalCheckPoints=10
MaxNoOfConcurrentTransaction5=5000
MaxNoOfConcurrentOperations=250000
NoOfFragmentLogFiles=500
TimeBetweenWatchDogCheck=600000

Please suggest
Sany

How to repeat:
set cluster for atleast 2 datanodes

insert values to ndbd tables through contineous loop

observe the result

Thank you for taking the time to write to us, but this is not a bug. Please double-check the documentation available at http://dev.mysql.com/doc/

Please check your cluster configuration, please read the following url and adjust accordingly; 
http://dev.mysql.com/doc/refman/5.1/en/mysql-cluster-config-lcp-params.html

Please provide your config.ini file for full review. Also, can you please use ndb_error_reporter tool to gather more information. I did not see the 4009 error.
http://dev.mysql.com/doc/refman/5.1/en/mysql-cluster-utilities-ndb-error-reporter.html

The configuration files are

" my.cnf  "  in data nodes
=========================
[client]				 
port	= 3306				 
socket	= /var/lib/mysql/mysql.sock	 
[mysqld]				 
port	= 3306				 
socket	= /var/lib/mysql/mysql.sock	 
datadir = /var/lib/mysql		 
back-log=75
skip-innodb
max_connections=500
skip-locking				 
key_buffer = 384M			
myisam_sort_buffer_size=64m
join_buffer_size=1M
max_allowed_packet = 64M			 
table_cache = 1800
thread_cache_size=384
wait_timeout=7200
connect_timeout=10
tmp_table_size=64M
max_heap_table_size=64M
max_connect_errors=1000
read_rnd_buffer_size=524288
bulk_insert_buffer_size=8M
sort_buffer_size = 2M			 
net_buffer_length = 8K			 
read_buffer_size = 1M			 
log-bin=mysql-bin			 
query_cache_limit=4M
query_cache_size=128M
query_cache_type=1
query_prealloc_size=65536
querry_alloc_block-size=131072

[mysqld_safe]
nice=-5
open_files_limit=8192

[mysqldump]				 
quick					 
max_allowed_packet = 16M		 
[mysql]					 
no-auto-rehash				 

[myisamchk]				 
key_buffer = 64M			 
sort_buffer_size = 64M			 
read_buffer = 16M 			 
write_buffer = 16M			 
  			 
[mysqlhotcopy]				 
interactive-timeout  			 
[mysqld]				 
ndbcluster 				 
ndb-connectstring=192.168.3.97 
[mysql_cluster] 			 
ndb-connectstring=192.168.3.97  

and "config.ini" in mgmd node is
================================

[NDBD DEFAULT]				
NoOfReplicas=2				
DataMemory=600M				
IndexMemory=200M	
RedoBuffer=300M

		
						
# TCP/IP options:				
[TCP DEFAULT]				
portnumber=2202				
						
# Management process options:		
[NDB_MGMD] 				
						
hostname=192.168.3.97  	# Hostname or IP address of MGM node		
datadir=/var/lib/mysql-cluster	# Directory for MGM node logfiles		

						
						
							
# Options for data node 1 :			
[NDBD]						
		# (one [NDBD] section per data node)	

RedoBuffer=500M
DataMemory=256M
IndexMemory=256M							
hostname=192.168.3.99	# Hostname or IPs 	
datadir=/var/lib/mysql	# path for data node's datafiles 
MaxNoOfAttributes=15000	
TimeBetweenLocalCheckPoints=6
TimeBetweenGlobalCheckPoints=10			
MaxNoOfTables=1000				
MaxNoOfOrderedIndexes=10000			
MaxNoOfUniqueHashIndexes=10000
MaxNoOfConcurrentTransactions=5000			
MaxNoOfConcurrentOperations=250000            	
TransactionBufferMemory=10M
NoOfFragmentLogFiles=300
TimeBetweenWatchDogCheck=600000
MaxNoOfLocalOperations=275000			
							
							
# Options for data node 2 :			
[NDBD]						
		# (one [NDBD] section per data node)	

RedoBuffer=500M
DataMemory=256M	
IndexMemory=256M					
hostname=192.168.3.100	# Hostname or IPs 	
datadir=/var/lib/mysql	# path for data node's datafiles 
MaxNoOfAttributes=15000				
TimeBetweenLocalCheckPoints=6
TimeBetweenGlobalCheckPoints=10
MaxNoOfTables=1000				
MaxNoOfOrderedIndexes=10000			
MaxNoOfUniqueHashIndexes=10000	
MaxNoOfConcurrentTransactions=5000		
MaxNoOfConcurrentOperations=250000
TransactionBufferMemory=10M
NoOfFragmentLogFiles=300
TimeBetweenWatchDogCheck=600000            	
MaxNoOfLocalOperations=275000			
							
	# SQL node options:				
[mysqld]					 
hostname=192.168.3.96		# SQL Node IP
[mysqld]
hostname=192.168.3.95		# SQL/Offline IP	 	 

How to repeat:
set cluster with atleast two data nodes

and try to insert contineously into ndbd tables

observe the result after some time

Please, try to repeat with a newer version, 5.1.22, and inform about the results.

No feedback was provided for this bug for over a month, so it is
being suspended automatically. If you are able to provide the
information that was originally requested, please do so and change
the status of the bug back to "Open".