Description:
When performing an ALTER command on the cluster is receive the following message from the mysql command prompt:
mysql> ALTER TABLE `xxx`.`table2` ADD INDEX `i_table2_timestamp`(`timestamp`);
ERROR 1297 (HY000): Got temporary error 4025 'Node failure caused abort of transaction' from NDBCLUSTER
mysql>
Message from the ndb_mgm console:
Node 22: Forced node shutdown completed. Caused by error 2303: 'System error, node killed during node restart by other node(Internal error, programming error or missing error message, please report a bug). Temporary error, restart node'.
How to repeat:
/etc/mysql/config.ini
[TCP DEFAULT]
SendBufferMemory=16M
ReceiveBufferMemory=16M
[NDB_MGMD DEFAULT]
PortNumber=1186
Datadir=/usr/local/mysql/data/
[NDB_MGMD]
Id=1
Hostname=192.168.0.80
ArbitrationRank=1
[NDB_MGMD]
Id=2
Hostname=192.168.0.81
ArbitrationRank=1
[NDBD DEFAULT]
NoOfReplicas=2
Datadir=/usr/local/mysql/data/
FileSystemPathDD=/usr/local/mysql/data/
#FileSystemPathUndoFiles=/usr/local/mysql/data/
#FileSystemPathDataFiles=/usr/local/mysql/data/
DataMemory=9G
IndexMemory=2G
LockPagesInMainMemory=0
MaxNoOfConcurrentOperations=100000
StringMemory=25
MaxNoOfTables=20000
MaxNoOfOrderedIndexes=10000
MaxNoOfUniqueHashIndexes=2500
MaxNoOfAttributes=120000
DiskCheckpointSpeedInRestart=100M
FragmentLogFileSize=256M
InitFragmentLogFiles=FULL
NoOfFragmentLogFiles=53
RedoBuffer=256M
TimeBetweenLocalCheckpoints=20
TimeBetweenGlobalCheckpoints=1000
TimeBetweenEpochs=100
#MemReportFrequency=30
#BackupReportFrequency=10
### Params for setting logging
LogLevelStartup=15
LogLevelShutdown=15
LogLevelCheckpoint=8
LogLevelNodeRestart=15
### Params for increasing Disk throughput
BackupMaxWriteSize=1M
BackupDataBufferSize=16M
BackupLogBufferSize=4M
BackupMemory=20M
#Reports indicates that odirect=1 can cause io errors (os err code 5) on some systems. You must test.
#ODirect=1
### Watchdog
TimeBetweenWatchdogCheckInitial=60000
### TransactionInactiveTimeout - should be enabled in Production
TransactionInactiveTimeout=30000
TransactionDeadlockDetectionTimeout=60000
### CGE 6.3 - REALTIME EXTENSIONS
#RealTimeScheduler=1
#SchedulerExecutionTimer=80
#SchedulerSpinTimer=40
### DISK DATA
#SharedGlobalMemory=384M
#read my blog how to set this:
#DiskPageBufferMemory=3072M
### Multithreading
MaxNoOfExecutionThreads=8
### Increasing the LongMessageBuffer b/c of a bug (20090903)
LongMessageBuffer=32M
BatchSizePerLocalScan=512
[NDBD]
Id=21
Hostname=192.168.0.84
### CGE 6.3 - REALTIME EXTENSIONS
### PLEASE NOTE THAT THE BELOW ONLY WORKS IF YOU HAVE >1 CORE.
### YOU SHOULD CHECK cat /proc/interrupts AND CHOOSE THE CPUs
### THAT GENERATE THE LEAST INTERRUPS. TYPICALLY THE CPU HANDLING
### THE INTERRUPTS FOR THE COMMUNICATION INTERFACE USED FOR THE DATA NODE SHOULD
### BE AVOIDED FOR THE LockExecuteThreadToCPU, BUT YOU CAN
### LockMaintThreadsToCPU TO THAT CPU SINCE IT DOES NOT AFFECT THE
### REALTIME ASPECTS (THIS IS TRUE FOR UP TO TWO DATA NODES ONE ONE COMPUTER.
#LockExecuteThreadToCPU=X
#LockMaintThreadsToCPU=Y
[NDBD]
Id=22
Hostname=192.168.0.85
### CGE 6.3 - REALTIME EXTENSIONS
### PLEASE NOTE THAT THE BELOW ONLY WORKS IF YOU HAVE >1 CORE.
### YOU SHOULD CHECK cat /proc/interrupts AND CHOOSE THE CPUs
### THAT GENERATE THE LEAST INTERRUPS. TYPICALLY THE CPU HANDLING
### THE INTERRUPTS FOR THE COMMUNICATION INTERFACE USED FOR THE DATA NODE SHOULD
### BE AVOIDED FOR THE LockExecuteThreadToCPU, BUT YOU CAN
### LockMaintThreadsToCPU TO THAT CPU SINCE IT DOES NOT AFFECT THE
### REALTIME ASPECTS (THIS IS TRUE FOR UP TO TWO DATA NODES ONE ONE COMPUTER.
#LockExecuteThreadToCPU=X
#LockMaintThreadsToCPU=Y
[NDBD]
Id=23
Hostname=192.168.0.86
### CGE 6.3 - REALTIME EXTENSIONS
### PLEASE NOTE THAT THE BELOW ONLY WORKS IF YOU HAVE >1 CORE.
### YOU SHOULD CHECK cat /proc/interrupts AND CHOOSE THE CPUs
### THAT GENERATE THE LEAST INTERRUPS. TYPICALLY THE CPU HANDLING
### THE INTERRUPTS FOR THE COMMUNICATION INTERFACE USED FOR THE DATA NODE SHOULD
### BE AVOIDED FOR THE LockExecuteThreadToCPU, BUT YOU CAN
### LockMaintThreadsToCPU TO THAT CPU SINCE IT DOES NOT AFFECT THE
### REALTIME ASPECTS (THIS IS TRUE FOR UP TO TWO DATA NODES ONE ONE COMPUTER.
#LockExecuteThreadToCPU=X
#LockMaintThreadsToCPU=Y
[NDBD]
Id=24
Hostname=192.168.0.87
### CGE 6.3 - REALTIME EXTENSIONS
### PLEASE NOTE THAT THE BELOW ONLY WORKS IF YOU HAVE >1 CORE.
### YOU SHOULD CHECK cat /proc/interrupts AND CHOOSE THE CPUs
### THAT GENERATE THE LEAST INTERRUPS. TYPICALLY THE CPU HANDLING
### THE INTERRUPTS FOR THE COMMUNICATION INTERFACE USED FOR THE DATA NODE SHOULD
### BE AVOIDED FOR THE LockExecuteThreadToCPU, BUT YOU CAN
### LockMaintThreadsToCPU TO THAT CPU SINCE IT DOES NOT AFFECT THE
### REALTIME ASPECTS (THIS IS TRUE FOR UP TO TWO DATA NODES ONE ONE COMPUTER.
#LockExecuteThreadToCPU=X
#LockMaintThreadsToCPU=Y
## BELOW ARE TWO (INACTIVE) SLOTS FOR DATA NODES TO ALLOW FOR GROWTH
#[NDBD]
#Id=7
#Hostname=
### CGE 6.3 - REALTIME EXTENSIONS
### PLEASE NOTE THAT THE BELOW ONLY WORKS IF YOU HAVE >1 CORE.
### YOU SHOULD CHECK cat /proc/interrupts AND CHOOSE THE CPUs
### THAT GENERATE THE LEAST INTERRUPS. TYPICALLY THE CPU HANDLING
### THE INTERRUPTS FOR THE COMMUNICATION INTERFACE USED FOR THE DATA NODE SHOULD
### BE AVOIDED FOR THE LockExecuteThreadToCPU, BUT YOU CAN
### LockMaintThreadsToCPU TO THAT CPU SINCE IT DOES NOT AFFECT THE
### REALTIME ASPECTS (THIS IS TRUE FOR UP TO TWO DATA NODES ONE ONE COMPUTER.
#LockExecuteThreadToCPU=X
#LockMaintThreadsToCPU=Y
#[NDBD]
#Id=8
#Hostname=
### CGE 6.3 - REALTIME EXTENSIONS
### PLEASE NOTE THAT THE BELOW ONLY WORKS IF YOU HAVE >1 CORE.
### YOU SHOULD CHECK cat /proc/interrupts AND CHOOSE THE CPUs
### THAT GENERATE THE LEAST INTERRUPS. TYPICALLY THE CPU HANDLING
### THE INTERRUPTS FOR THE COMMUNICATION INTERFACE USED FOR THE DATA NODE SHOULD
### BE AVOIDED FOR THE LockExecuteThreadToCPU, BUT YOU CAN
### LockMaintThreadsToCPU TO THAT CPU SINCE IT DOES NOT AFFECT THE
### REALTIME ASPECTS (THIS IS TRUE FOR UP TO TWO DATA NODES ONE ONE COMPUTER.
#LockExecuteThreadToCPU=X
#LockMaintThreadsToCPU=Y
[MYSQLD DEFAULT]
BatchSize=512
#BatchByteSize=2048K
#MaxScanBatchSize=2048K
[MYSQLD]
Id=11
Hostname=192.168.0.81
[MYSQLD]
Id=12
Hostname=192.168.0.81
[MYSQLD]
Id=13
Hostname=192.168.0.81
[MYSQLD]
Id=14
Hostname=192.168.0.82
[MYSQLD]
Id=15
Hostname=192.168.0.82
[MYSQLD]
Id=16
Hostname=192.168.0.82
### SLOTS (one for each ndb_mgmd) FOR HELPER APPLICATIONS SUCH AS ndb_show_tables etc
[MYSQLD]
Hostname=192.168.0.80
[MYSQLD]
Hostname=192.168.0.81