MySQL Bugs: #46061: Cluster with more than 8 MySql/API nodes

Bug #46061	Cluster with more than 8 MySql/API nodes
Submitted:	8 Jul 2009 21:09	Modified:	15 Sep 2009 13:44
Reporter:	Sajjad Tariq	Email Updates:
Status:	In progress	Impact on me:	None
Category:	MySQL Cluster: Cluster (NDB) storage engine	Severity:	S1 (Critical)
Version:	mysql-5.1-telco-7.0	OS:	Windows (XP)
Assigned to:	Assigned Account	CPU Architecture:	Any
Tags:	MySql/API nodes 5.1.34-ndb-7.0.6-cluster-gpl, MySql/API nodes 5.1.35-ndb-7.0.7-cluster-gpl

Description:
When ndb_mgmd is started on a Cluster with more than 8 MySql/API nodes, then ndb_mgm and the storage NDBD can not get the config and ndb_mgmd crashes. The same setup works for 8 MySql/API nodes.

=====================================================================

I start up the management node with following Config.ini

C:\Program Files\MySQL\MySQL Server 7.0\bin>ndb_mgmd --initial
2009-07-08 14:45:10 [MgmSrvr] INFO     -- NDB Cluster Management Server. mysql-5
.1.34 ndb-7.0.6
2009-07-08 14:45:10 [MgmSrvr] INFO     -- Reading cluster configuration from 'C:
/mysql/mysql-cluster/ndb_mgmd/config.ini'

=====================================================================

When I start the management console I get unhandled exception error which my Visual Studio Just-In-Time Debugger tries to debug.

=====================================================================
On the management console I get this out put

C:\Program Files\MySQL\MySQL Server 7.0\bin>ndb_mgm
-- NDB Cluster -- Management Client --
ndb_mgm> show
Connected to Management Server at: MGMD1:1186
Failed to unpack buffer
Could not get configuration
*     0: No error
*        Executing: ndb_mgm_get_configuration
ndb_mgm>

How to repeat:
Config.ini
==========

[mysqld default]
[mysqld]
id=50
[mysqld]
id=51
[mysqld]
id=52
[mysqld]
id=53
[mysqld]
id=54
[mysqld]
id=55
[mysqld]
id=56
[mysqld]
id=57
[mysqld]
id=58
[mysqld]
id=59

[tcp default]
SendBufferMemory=2M
ReceiveBufferMemory=2M

# Increasing the sizes of these 2 buffers beyond the default values
# helps prevent bottlenecks due to slow disk I/O.

[ndb_mgmd default]
datadir=C:/mysql/mysql-cluster/ndb_mgmd

[ndb_mgmd]
id=1
hostname=mgmd1

#[ndb_mgmd]
#id=2
#hostname=mgmd2

[ndbd default]
noofreplicas=2
#MaxNoOfUniqueHashIndexes=4294967039
#MaxNoOfUniqueHashIndexes=4294967000
#MaxNoOfUniqueHashIndexes=1000000

MaxNoOfOrderedIndexes=10000
MaxNoOfAttributes=10000

TransactionDeadlockDetectionTimeout=12000

StopOnError=0

#MaxNoOfAttributes=4294967039
#datadir=d:/mysql/ndbd
datadir=C:/mysql/mysql-cluster/ndbd

#DataMemory=3072M
DataMemory=1280M
IndexMemory=150M
#IndexMemory=384M

# The values provided for DataMemory and IndexMemory assume 4 GB RAM
# per data node. However, for best results, you should first calculate
# the memory that would be used based on the data you actually plan to
# store (you may find the ndb_size.pl utility helpful in estimating
# this), then allow an extra 20% over the calculated values. Naturally,
# you should ensure that each data node host has at least as much
# physical memory as the sum of these two values.

 ODirect=1

# Enabling this parameter causes NDBCLUSTER to try using O_DIRECT
# writes for local checkpoints and redo logs; this can reduce load on
# CPUs. We recommend doing so when using MySQL Cluster NDB 6.2.3 or
# newer on systems running Linux kernel 2.6 or later.

#NoOfFragmentLogFiles=300

MaxNoOfConcurrentOperations=100000
#MaxNoOfConcurrentOperations=250000

#SchedulerSpinTimer=400
#SchedulerExecutionTimer=100
#RealTimeScheduler=1
# Setting these parameters allows you to take advantage of real-time scheduling
# of NDBCLUSTER threads (introduced in MySQL Cluster NDB 6.3.4) to get higher
# throughput.

UndoIndexBuffer=10M
TimeBetweenLocalCheckpoints=20

#TimeBetweenGlobalCheckpoints=1000
#TimeBetweenEpochs=200
#DiskCheckpointSpeed=10M
#DiskCheckpointSpeedInRestart=100M
#RedoBuffer=32M

# CompressedLCP=1
# CompressedBackup=1
# Enabling CompressedLCP and CompressedBackup causes, respectively, local checkpoint files 

and backup files to be compressed, which can result in a space savings of up to 50% over 

noncompressed LCPs and backups.

[ndbd]
id=31
hostname=ndbd204

[ndbd]
id=32
hostname=ndbd203

[ndbd]
id=33
hostname=ndbd202

[ndbd]
id=34
hostname=ndbd201

=====================================================================

my.ini
==========
[mysql_cluster]
ndb-connectstring=MGMD1

[mysqld]
basedir="C:/Program Files/MySQL/MySQL Server 7.0/"
datadir=C:/mysql/mysql-cluster/data/

port=3306
default-character-set=latin1
default-storage-engine=ndbcluster
skip-innodb
ndbcluster
ndb-use-exact-count=0
ndb-index-stat-enable=0
ndb-force-send=1
engine-condition-pushdown=1

max_allowed_packet=16M
delayed_insert_timeout=10000
connect_timeout=100000

[ndb_mgmd]
config-file="C:/mysql/mysql-cluster/ndb_mgmd/config.ini"
configdir="C:/mysql/mysql-cluster/ndb_mgmd/"

[ndbd default]

[ndbd]

[ndb_mgm]

Suggested fix:
Possible problem in ndb_mgm_get_configuration.

I don't fully understand how this is different from your bug #45733?

In Bug #45733 the problem is number of storage nodes, this bug is about number of MySql/API nodes.

After installing the 7.0.7 patch, mgmd can confirm the settings and mgm can display them, but data node is not starting. With 8 MySql/API nodes same config works.

Config
======

[TCP DEFAULT]
SendBufferMemory=2M
ReceiveBufferMemory=2M

[NDB_MGMD DEFAULT]
PortNumber=1186
datadir=C:/mysql/mysql-cluster/ndb_mgmd

[ndb_mgmd]
id=1
hostname=mgmd1
ArbitrationRank=1

#[NDB_MGMD]
#Id=2
#Hostname=mgmd2
#ArbitrationRank=1

[NDBD DEFAULT]
NoOfReplicas=2
Datadir=C:/mysql/mysql-cluster/ndbd/
#DataMemory=1280M
#IndexMemory=150M
DataMemory=512M
IndexMemory=62M
LockPagesInMainMemory=0

MaxNoOfConcurrentOperations=100000

#StringMemory=25
#MaxNoOfTables=4096
MaxNoOfOrderedIndexes=10000
MaxNoOfAttributes=10000
#MaxNoOfUniqueHashIndexes=512

#DiskCheckpointSpeedInRestart=100M
#FragmentLogFileSize=256M
#InitFragmentLogFiles=FULL
#NoOfFragmentLogFiles=6
#RedoBuffer=32M

TimeBetweenLocalCheckpoints=20
TimeBetweenGlobalCheckpoints=1000
TimeBetweenEpochs=100

MemReportFrequency=30
BackupReportFrequency=10

### Params for setting logging 
LogLevelStartup=15
LogLevelShutdown=15
LogLevelCheckpoint=8
LogLevelNodeRestart=15

### Params for increasing Disk throughput 
BackupMaxWriteSize=1M
BackupDataBufferSize=16M
BackupLogBufferSize=4M
BackupMemory=20M
#Reports indicates that odirect=1 can cause io errors (os err code 5) on some systems. You 

must test.
ODirect=1

### Watchdog 
TimeBetweenWatchdogCheckInitial=30000

### TransactionInactiveTimeout  - should be enabled in Production 
TransactionInactiveTimeout=30000
### CGE 6.3 - REALTIME EXTENSIONS 
#RealTimeScheduler=1
#SchedulerExecutionTimer=80
#SchedulerSpinTimer=40
### DISK DATA 
SharedGlobalMemory=384M
#read my blog how to set this:
#DiskPageBufferMemory=3072M
BatchSizePerLocalScan=512

[ndbd]
id=21
hostname=ndb214

[ndbd]
id=22
hostname=ndb213

[ndbd]
id=23
hostname=ndb212

[ndbd]
id=24
hostname=ndb211

[ndbd]
id=25
hostname=ndb210

#[ndbd]
#id=26
#hostname=ndb209

#[ndbd]
#id=27
#hostname=ndb208

[ndbd]
id=28
hostname=ndb207

[ndbd]
id=29
hostname=ndb206

[ndbd]
id=30
hostname=ndb205

[ndbd]
id=31
hostname=ndb204

[ndbd]
id=32
hostname=ndb203

[ndbd]
id=33
hostname=ndb202

[ndbd]
id=34
hostname=ndb201

[MYSQLD DEFAULT]
BatchSize=512
#BatchByteSize=2048K
#MaxScanBatchSize=2048K

[mysqld]
id=50
[mysqld]
id=51
[mysqld]
id=52
[mysqld]
id=53
[mysqld]
id=54
[mysqld]
id=55
[mysqld]
id=56
[mysqld]
id=57
[mysqld]
id=58
[mysqld]
id=59
[mysqld]
id=60
[mysqld]
id=61
[mysqld]
id=62
[mysqld]
id=63
[mysqld]
id=64
[mysqld]
id=65
[mysqld]
id=66
[mysqld]
id=67
[mysqld]
id=68
[mysqld]
id=69
[mysqld]
id=70
[mysqld]
id=71
[mysqld]
id=72
[mysqld]
id=73
[mysqld]
id=74
[mysqld]
id=75
[mysqld]
id=76
[mysqld]
id=77
[mysqld]
id=78
[mysqld]
id=79
[mysqld]
id=80
[mysqld]
id=81
[mysqld]
id=82
[mysqld]
id=83
[mysqld]
id=84
[mysqld]
id=85
[mysqld]
id=86
[mysqld]
id=87
[mysqld]
id=88
[mysqld]
id=89
[mysqld]
id=90
[mysqld]
id=91
[mysqld]
id=92
[mysqld]
id=93
[mysqld]
id=94
[mysqld]
id=95
[mysqld]
id=96
[mysqld]
id=97
[mysqld]
id=98
[mysqld]
id=99
[mysqld]
id=100
[mysqld]
id=101
[mysqld]
id=102
[mysqld]
id=103
[mysqld]
id=104
[mysqld]
id=105
[mysqld]
id=106
[mysqld]
id=107
[mysqld]
id=108
[mysqld]
id=109
[mysqld]
id=110
[mysqld]
id=111
[mysqld]
id=112
[mysqld]
id=113
[mysqld]
id=114
[mysqld]
id=115
[mysqld]
id=116
[mysqld]
id=117
[mysqld]
id=118
[mysqld]
id=119
[mysqld]
id=120
[mysqld]
id=121
[mysqld]
id=122
[mysqld]
id=123
[mysqld]
id=124
[mysqld]
id=125
[mysqld]
id=126
[mysqld]
id=127
[mysqld]
id=128
[mysqld]
id=129
[mysqld]
id=130
[mysqld]
id=131
[mysqld]
id=132
[mysqld]
id=133
[mysqld]
id=134
[mysqld]
id=135
[mysqld]
id=136
[mysqld]
id=137
[mysqld]
id=138
[mysqld]
id=139
[mysqld]
id=140
[mysqld]
id=141
[mysqld]
id=142
[mysqld]
id=143
[mysqld]
id=144
[mysqld]
id=145
[mysqld]
id=146
[mysqld]
id=147
[mysqld]
id=148
[mysqld]
id=149
[mysqld]
id=150
[mysqld]
id=151
[mysqld]
id=152
[mysqld]
id=153
[mysqld]
id=154
[mysqld]
id=155
[mysqld]
id=156
[mysqld]
id=157
[mysqld]
id=158
[mysqld]
id=159
[mysqld]
id=160
[mysqld]
id=161
[mysqld]
id=162
[mysqld]
id=163
[mysqld]
id=164
[mysqld]
id=165
[mysqld]
id=166
[mysqld]
id=167
[mysqld]
id=168
[mysqld]
id=169
[mysqld]
id=170
[mysqld]
id=171
[mysqld]
id=172
[mysqld]
id=173
[mysqld]
id=174
[mysqld]
id=175
[mysqld]
id=176
[mysqld]
id=177
[mysqld]
id=178
[mysqld]
id=179
[mysqld]
id=180
[mysqld]
id=181
[mysqld]
id=182
[mysqld]
id=183
[mysqld]
id=184
[mysqld]
id=185
[mysqld]
id=186
[mysqld]
id=187
[mysqld]
id=188
[mysqld]
id=189
[mysqld]
id=190
[mysqld]
id=191
[mysqld]
id=192
[mysqld]
id=193
[mysqld]
id=194
[mysqld]
id=195
[mysqld]
id=196
[mysqld]
id=197
[mysqld]
id=198
[mysqld]
id=199
[mysqld]
id=200
[mysqld]
id=201
[mysqld]
id=202
[mysqld]
id=203
[mysqld]
id=204
[mysqld]
id=205
[mysqld]
id=206
[mysqld]
id=207
[mysqld]
id=208
[mysqld]
id=209
[mysqld]
id=210
[mysqld]
id=211
[mysqld]
id=212
[mysqld]
id=213
[mysqld]
id=214
[mysqld]
id=215
[mysqld]
id=216
[mysqld]
id=217
[mysqld]
id=218
[mysqld]
id=219
[mysqld]
id=220
[mysqld]
id=221
[mysqld]
id=222
[mysqld]
id=223
[mysqld]
id=224
[mysqld]
id=225
[mysqld]
id=226
[mysqld]
id=227
[mysqld]
id=228
[mysqld]
id=229
[mysqld]
id=230
[mysqld]
id=231
[mysqld]
id=232
[mysqld]
id=233
[mysqld]
id=234
[mysqld]
id=235
[mysqld]
id=236
[mysqld]
id=237
[mysqld]
id=238
[mysqld]
id=239
[mysqld]
id=240
[mysqld]
id=241
[mysqld]
id=242
[mysqld]
id=243
[mysqld]
id=244
[mysqld]
id=245
[mysqld]
id=246
[mysqld]
id=247
[mysqld]
id=248
[mysqld]
id=249

For some reason mgmd reads the config file twice. With 8 API nodes it only reads one.

C:\Program Files\MySQL\MySQL Server 7.0\bin>ndb_mgmd --initial
2009-07-17 09:22:54 [MgmSrvr] INFO     -- NDB Cluster Management Server. mysql-5
.1.35 ndb-7.0.7
2009-07-17 09:22:54 [MgmSrvr] INFO     -- Reading cluster configuration from 'C:
/mysql/mysql-cluster/ndb_mgmd/config.ini'
2009-07-17 09:23:00 [MgmSrvr] INFO     -- Reading cluster configuration from 'C:
/mysql/mysql-cluster/ndb_mgmd/config.ini'

=============================================================================
Mgm console displays this output.

C:\Program Files\MySQL\MySQL Server 7.0\bin>ndb_mgm
-- NDB Cluster -- Management Client --
ndb_mgm> show
Connected to Management Server at: localhost:1186
Cluster Configuration
---------------------
[ndbd(NDB)]     12 node(s)
id=21 (not connected, accepting connect from ndb214)
id=22 (not connected, accepting connect from ndb213)
id=23 (not connected, accepting connect from ndb212)
id=24 (not connected, accepting connect from ndb211)
id=25 (not connected, accepting connect from ndb210)
id=28 (not connected, accepting connect from ndb207)
id=29 (not connected, accepting connect from ndb206)
id=30 (not connected, accepting connect from ndb205)
id=31 (not connected, accepting connect from ndb204)
id=32 (not connected, accepting connect from ndb203)
id=33 (not connected, accepting connect from ndb202)
id=34 (not connected, accepting connect from ndb201)

[ndb_mgmd(MGM)] 1 node(s)
id=1    @MGMD1  (mysql-5.1.35 ndb-7.0.7)

[mysqld(API)]   200 node(s)
id=50 (not connected, accepting connect from any host)
id=51 (not connected, accepting connect from any host)
id=52 (not connected, accepting connect from any host)
id=53 (not connected, accepting connect from any host)
id=54 (not connected, accepting connect from any host)
id=55 (not connected, accepting connect from any host)
id=56 (not connected, accepting connect from any host)
id=57 (not connected, accepting connect from any host)
id=58 (not connected, accepting connect from any host)
id=59 (not connected, accepting connect from any host)
:
:
:
id=246 (not connected, accepting connect from any host)
id=247 (not connected, accepting connect from any host)
id=248 (not connected, accepting connect from any host)
id=249 (not connected, accepting connect from any host)

ndb_mgm>

NDBD tries to start but crashes with ‘—-initial’ command.

C:\Program Files\MySQL\MySQL Server 7.0\bin>ndbd --initial
2009-07-17 09:28:13 [ndbd] INFO     -- Configuration fetched from 'mgmd1:
1186', generation: 1
2009-07-17 09:28:13 [ndbd] INFO     -- Ndb started
NDBMT: non-mt
2009-07-17 09:28:14 [ndbd] INFO     -- NDB Cluster -- DB node 34
2009-07-17 09:28:14 [ndbd] INFO     -- mysql-5.1.35 ndb-7.0.7 --
2009-07-17 09:28:14 [ndbd] INFO     -- WatchDog timer is set to 6000 ms
2009-07-17 09:28:14 [ndbd] INFO     -- Ndbd_mem_manager::init(1) min: 516Mb init
ial: 900Mb
Adding 729Mb to ZONE_LO (1,23327)
Adding 171Mb to ZONE_LO (23780,5472)
2009-07-17 09:29:13 [ndbd] INFO     -- Start initiated (mysql-5.1.35 ndb-7.0.7)
2009-07-17 09:29:13 [ndbd] WARNING  -- Ndb kernel thread 0 is stuck in: Job Hand
ling elapsed=109
2009-07-17 09:29:16 [ndbd] INFO     -- Watchdog: User time: 12812500  System tim
e: 7812500
NDBFS: Created new file thread 0
NDBFS: Created new file thread 1
NDBFS: Created new file thread 2
NDBFS: Created new file thread 3
NDBFS: Created new file thread 4
NDBFS: Created new file thread 5
NDBFS: Created new file thread 6
NDBFS: Created new file thread 7
NDBFS: Created new file thread 8
NDBFS: Created new file thread 9
NDBFS: Created new file thread 10
NDBFS: Created new file thread 11
NDBFS: Created new file thread 12
NDBFS: Created new file thread 13
NDBFS: Created new file thread 14
NDBFS: Created new file thread 15
NDBFS: Created new file thread 16
NDBFS: Created new file thread 17
NDBFS: Created new file thread 18
NDBFS: Created new file thread 19
NDBFS: Created new file thread 20
NDBFS: Created new file thread 21
NDBFS: Created new file thread 22
NDBFS: Created new file thread 23
NDBFS: Created new file thread 24
NDBFS: Created new file thread 25
NDBFS: Created new file thread 26
2009-07-17 09:29:16 [ndbd] INFO     -- Watchdog: User time: 12812500  System tim
e: 7812500
2009-07-17 09:29:16 [ndbd] WARNING  -- Watchdog: Warning overslept 3093 ms, expe
cted 100 ms.
WOPool::init(61, 9)
RWPool::init(22, 13)
2009-07-17 09:29:17 [ndbd] INFO     -- Watchdog: User time: 12812500  System tim
e: 7968750
2009-07-17 09:29:19 [ndbd] WARNING  -- Watchdog: Warning overslept 1093 ms, expe
cted 100 ms.
2009-07-17 09:29:19 [ndbd] INFO     -- Watchdog: User time: 12968750  System tim
e: 7968750
2009-07-17 09:29:19 [ndbd] WARNING  -- Watchdog: Warning overslept 1718 ms, expe
cted 100 ms.
2009-07-17 09:29:19 [ndbd] INFO     -- DBACC could not allocate memory for param
eter IndexMemory
2009-07-17 09:29:19 [ndbd] INFO     -- Requested: 8192x7968 = 65273856 bytes
error=2327
2009-07-17 09:29:19 [ndbd] INFO     -- Error handler startup shutting down syste
m
2009-07-17 09:29:19 [ndbd] INFO     -- Error handler shutdown completed - exitin
g
2009-07-17 09:29:19 [ndbd] ALERT    -- Node 34: Forced node shutdown completed.
Occured during startphase 0.

C:\Program Files\MySQL\MySQL Server 7.0\bin>

=============================================================================
After the crash I tried to start the NDBD with out “--initial”

C:\Program Files\MySQL\MySQL Server 7.0\bin>ndbd
2009-07-17 09:31:47 [ndbd] INFO     -- Configuration fetched from 'mgmd1:
1186', generation: 1
2009-07-17 09:31:47 [ndbd] INFO     -- Ndb started
NDBMT: non-mt
2009-07-17 09:31:47 [ndbd] INFO     -- NDB Cluster -- DB node 34
2009-07-17 09:31:47 [ndbd] INFO     -- mysql-5.1.35 ndb-7.0.7 --
2009-07-17 09:31:47 [ndbd] INFO     -- WatchDog timer is set to 6000 ms
2009-07-17 09:31:47 [ndbd] INFO     -- Ndbd_mem_manager::init(1) min: 516Mb init
ial: 900Mb
2009-07-17 09:31:49 [ndbd] INFO     -- Adding 729Mb to ZONE_LO (1,23327)Watchdog
: User time: 12968750  System time: 5468750

Adding 171Mb to ZONE_LO (23780,5472)
2009-07-17 09:31:54 [ndbd] WARNING  -- Watchdog: Warning overslept 625 ms, expec
ted 100 ms.
2009-07-17 09:31:54 [ndbd] INFO     -- Watchdog: User time: 13125000  System tim
e: 6250000
2009-07-17 09:31:54 [ndbd] WARNING  -- Watchdog: Warning overslept 4812 ms, expe
cted 100 ms.
2009-07-17 09:31:54 [ndbd] INFO     -- Watchdog: User time: 13125000  System tim
e: 6250000
2009-07-17 09:31:54 [ndbd] WARNING  -- Watchdog: Warning overslept 578 ms, expec
ted 100 ms.
2009-07-17 09:32:37 [ndbd] INFO     -- Start initiated (mysql-5.1.35 ndb-7.0.7)
2009-07-17 09:32:38 [ndbd] WARNING  -- Ndb kernel thread 0 is stuck in: Job Hand
ling elapsed=109
2009-07-17 09:32:40 [ndbd] INFO     -- Watchdog: User time: 13125000  System tim
e: 7500000
NDBFS: Created new file thread 0
NDBFS: Created new file thread 1
NDBFS: Created new file thread 2
NDBFS: Created new file thread 3
NDBFS: Created new file thread 4
NDBFS: Created new file thread 5
NDBFS: Created new file thread 6
NDBFS: Created new file thread 7
NDBFS: Created new file thread 8
NDBFS: Created new file thread 9
NDBFS: Created new file thread 10
NDBFS: Created new file thread 11
NDBFS: Created new file thread 12
NDBFS: Created new file thread 13
NDBFS: Created new file thread 14
NDBFS: Created new file thread 15
NDBFS: Created new file thread 16
NDBFS: Created new file thread 17
NDBFS: Created new file thread 18
NDBFS: Created new file thread 19
NDBFS: Created new file thread 20
NDBFS: Created new file thread 21
NDBFS: Created new file thread 22
NDBFS: Created new file thread 23
NDBFS: Created new file thread 24
NDBFS: Created new file thread 25
NDBFS: Created new file thread 26
2009-07-17 09:32:40 [ndbd] INFO     -- Watchdog: User time: 13125000  System tim
e: 7500000
2009-07-17 09:32:40 [ndbd] WARNING  -- Watchdog: Warning overslept 2531 ms, expe
cted 100 ms.
WOPool::init(61, 9)
RWPool::init(22, 13)
2009-07-17 09:32:43 [ndbd] WARNING  -- Ndb kernel thread 0 is stuck in: Print Jo
b Buffers at crash elapsed=109
2009-07-17 09:32:43 [ndbd] INFO     -- Watchdog: User time: 13437500  System tim
e: 7500000
2009-07-17 09:32:43 [ndbd] INFO     -- DBACC could not allocate memory for param
eter IndexMemory
2009-07-17 09:32:43 [ndbd] INFO     -- Requested: 8192x7968 = 65273856 bytes
error=2327
2009-07-17 09:32:43 [ndbd] INFO     -- Error handler startup shutting down syste
m
2009-07-17 09:32:43 [ndbd] INFO     -- Error handler shutdown completed - exitin
g
2009-07-17 09:32:43 [ndbd] ALERT    -- Node 34: Forced node shutdown completed.
Occured during startphase 0.

C:\Program Files\MySQL\MySQL Server 7.0\bin>
=============================================================================

Now same settings works and I got a working cluster with only 8 MySql/API nodes. 

I am also attaching the logs files from the NDBD.

ndb_34_error.log

Attachment: ndb_34_error.log (application/octet-stream, text), 1.07 KiB.

ndb_34_trace.log.1

Attachment: ndb_34_trace.log.1 (text/plain), 15.29 KiB.

ndb_34_trace.log.2

Attachment: ndb_34_trace.log.2 (text/plain), 15.29 KiB.

A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/79071

2952 jack andrews	2009-07-21 [merge]
      merge Bug #46061

hi Sajjad,

i've tried to reproduce you error different ways.  first a couple of notes:

  1) ndbd --initial sometimes fails due to a bug where empty directories cannot be deleted.  we have a patch waiting approval.  for your --initial tests, can you make sure you remove the fs dir first 
  eg. rm -rf c:/eight-api-nodes/data/ndb_34_fs
if you have cygwin installed

  2) i see the repeated "reading config" log lines and i can't explain that right now.  i am still investigating, but this could be a bug or a feature.  we write a binary config and the second message could be ndb_mgmd reading that new config.

  3) i can connect at least one ndbd to the cluster, and then i configured ndb with low memory requirements so i could connect many ndbds on one machine.  i am able to connect a number of data nodes.  i'll attach my config after this post.

then i looked at the ndb_34_error.log you attached.  it looks like you might need to reduce your config too.  look at my config in the next attachment for tips.

i used:
  ndb_mgmd.exe -f ini\eight-plus.ini --reload --initial --nodaemon
  ndbd.exe --initial --nodaemon

ta, jack.

this ini file runs for me -- have a look at the config variables in this file for hints to reduce your memory on ndbd's

Attachment: eight-plus.ini (application/octet-stream, text), 4.06 KiB.

Jack,

Thank you for looking into this updated bug report. The settings you tested are very low, 
DataMemory=10M
IndexMemory=5M

I am trying to load over a GB of data on to twelve data nodes. What concerns me is that why ndb has to request so much more with more then eight MySQL/API nodes. Following is the error I was hopping we could look into to make sure it is acting appropriately.

Error object: Requested: 8192x7968 = 65273856 bytes

Thank you,

Sajjad

> Thank you for looking into this updated bug report. The settings you
> tested are very low, 
> DataMemory=10M
> IndexMemory=5M

yes, but can you try this setting just to see if the cluster 
starts ok?  

we think we know the problem and that is a difference between 
linux and windows memory management/allocator routines.  for 
larger blocks of memory, linux mmap()s, but windows only 
grabs memory from the heap.  

a solution is to use VirtualAlloc() instead of new/malloc on
windows for larger requests.  VirtualAlloc() is the windows
equivalent of an anonymous, private mmap on linux.

> I am trying to load over a GB of data on to twelve data nodes. What
> concerns me is that why ndb has to request so much more with more then
> eight MySQL/API nodes. Following is the error I was hopping we could
> look into to make sure it is acting appropriately.
> 
> Error object: Requested: 8192x7968 = 65273856 bytes

i'll look into this error.

ta, jack.

>> Error object: Requested: 8192x7968 = 65273856 bytes
>
> i'll look into this error.

this confirms my suspicions.  this error is printed in
  SimulatedBlock::allocRecordAligned
and is printed when malloc() returns 0.

Sajjad,

> When ndb_mgmd is started on a Cluster with more than 
> 8 MySql/API nodes

just a query on why you need so many API nodes?  have 
you tested having 8 API nodes compared to, say, 4
API nodes?

Jack,

Sorry for taking this long to reply back, we were making some major changes in our back and front end.

Well the configuration works for 8 API nodes, but here is another problem, third API doesn’t discover schemas from the cluster.

Thank you,

Sajjad

mysqld.err
================================================================================
090812 15:03:41 [Note] Plugin 'InnoDB' disabled by command line option
090812 15:03:42 [Note] NDB: NodeID is 15, management server 'MGMD1:1186'
090812 15:03:44 [Note] NDB[0]: NodeID: 15, some storage nodes connected
090812 15:03:44 [Note] Starting Cluster Binlog Thread
090812 15:03:44 [Note] Event Scheduler: Loaded 0 events
090812 15:03:44 [Note] mysqld: ready for connections.
Version: '5.1.34-ndb-7.0.6-cluster-gpl'  socket: ''  port: 3306  MySQL Cluster Server (GPL)
090812 15:03:45 [Note] NDB Binlog: DISCOVER TABLE Event: REPL$mysql/ndb_schema

Jack,

Sorry it took this long to respond, with 7.0.7 GA release I am still unable to get the third API node to discover schemas. MGM console shows the API making the connection but API never gets the tables/schema.

Thank you,

Sajjad 

===============================================================================
090915  7:40:13 [Note] Plugin 'InnoDB' disabled by command line option
090915  7:40:13 [Note] NDB: NodeID is 15, management server 'MGMD1:1186'
090915  7:40:15 [Note] NDB[0]: NodeID: 15, some storage nodes connected
090915  7:40:15 [Warning] NDB: server id set to zero will cause any other mysqld with bin log to log with wrong server id
090915  7:40:15 [Note] Starting Cluster Binlog Thread
090915  7:40:17 [Note] Event Scheduler: Loaded 0 events
090915  7:40:17 [Note] mysqld: ready for connections.
Version: '5.1.35-ndb-7.0.7-cluster-gpl-log'  socket: ''  port: 3306  MySQL Cluster Server (GPL)
090915  7:40:18 [Note] NDB Binlog: DISCOVER TABLE Event: REPL$mysql/ndb_schema