MySQL Bugs: #16447: partitioned database when start partial

Bug #16447	partitioned database when start partial
Submitted:	12 Jan 2006 11:43	Modified:	12 Apr 2006 0:01
Reporter:	Tomas Ulin	Email Updates:
Status:	Closed	Impact on me:	None
Category:	MySQL Cluster: Cluster (NDB) storage engine	Severity:	S3 (Non-critical)
Version:	4.1->	OS:	Any
Assigned to:	Jonas Oreland	CPU Architecture:	Any

Description:
2-node cluster is busy handling a node failure
node is restarted with low partition timeout
and desides to start by itself
cluster comes up partitioned

2006-01-10 14:20:35 [MgmSrvr] ALERT    -- Node 3: Node 4 Disconnected
2006-01-10 14:20:35 [MgmSrvr] INFO     -- Node 3: Communication to Node 4 closed
2006-01-10 14:20:35 [MgmSrvr] ALERT    -- Node 3: Network partitioning - arbitration required
2006-01-10 14:20:35 [MgmSrvr] INFO     -- Node 3: President restarts arbitration thread [state=7]
2006-01-10 14:20:35 [MgmSrvr] INFO     -- Node 1: Node 4 Connected
2006-01-10 14:20:35 [MgmSrvr] ALERT    -- Node 3: Arbitration won - positive reply from node 1
2006-01-10 14:20:35 [MgmSrvr] INFO     -- Node 3: GCP Take over started
2006-01-10 14:20:35 [MgmSrvr] INFO     -- Node 3: GCP Take over completed
2006-01-10 14:20:35 [MgmSrvr] INFO     -- Node 4: Node shutdown completed, restarting.
2006-01-10 14:20:35 [MgmSrvr] WARNING  -- Allocate nodeid (4) failed. Connection from ip 172.30.16.130. Returned error string "Id 4 already allocated by an
other node."
2006-01-10 14:20:35 [MgmSrvr] INFO     -- Mgmt server state: node id's  2 3 4 7 8 connected but not reserved
2006-01-10 14:20:35 [MgmSrvr] INFO     -- Mgmt server state: node id's  1 not connected but reserved
2006-01-10 14:20:35 [MgmSrvr] INFO     -- Node 3: Started arbitrator node 1 [ticket=03920004b4799a0b]
2006-01-10 14:20:38 [MgmSrvr] INFO     -- Mgmt server state: nodeid 4 reserved for ip 172.30.16.130, m_reserved_nodes 0000000000000672.
2006-01-10 14:20:38 [MgmSrvr] INFO     -- Node 1: Node 4 Connected
2006-01-10 14:20:39 [MgmSrvr] INFO     -- Mgmt server state: nodeid 4 freed, m_reserved_nodes 0000000000000662.
2006-01-10 14:20:50 [MgmSrvr] INFO     -- Node 4: Start phase 1 completed 
2006-01-10 14:20:56 [MgmSrvr] INFO     -- Node 3: Backup 10 started from node 1
2006-01-10 14:21:10 [MgmSrvr] INFO     -- Node 4: Start phase 2 completed (system restart)
2006-01-10 14:21:10 [MgmSrvr] INFO     -- Node 4: Start phase 3 completed (system restart)
2006-01-10 14:21:33 [MgmSrvr] WARNING  -- Node 3: Failure handling of node 4 has not completed in 1 min. - state = 3

....

2006-01-10 14:21:41 [MgmSrvr] INFO     -- Node 3: Communication to Node 4 opened
2006-01-10 14:21:41 [MgmSrvr] INFO     -- Node 3: Node 4 Connected
2006-01-10 14:21:41 [MgmSrvr] INFO     -- Node 4: Node 3 Connected
2006-01-10 14:21:53 [MgmSrvr] INFO     -- Node 4: Start phase 4 completed (system restart)
2006-01-10 14:21:54 [MgmSrvr] INFO     -- Node 4: Local checkpoint 120 started. Keep GCI = 172315 oldest restorable GCI = 172391
2006-01-10 14:24:39 [MgmSrvr] INFO     -- Mgmt server state: nodeid 11 reserved for ip 172.30.16.1, m_reserved_nodes 0000000000000e62.
2006-01-10 14:24:39 [MgmSrvr] INFO     -- Node 3: Node 11 Connected
2006-01-10 14:24:39 [MgmSrvr] INFO     -- Node 3: Node 11: API version 5.1.2
2006-01-10 14:24:39 [MgmSrvr] ALERT    -- Node 3: Node 11 Disconnected
2006-01-10 14:24:39 [MgmSrvr] INFO     -- Node 3: Communication to Node 11 closed
2006-01-10 14:24:41 [MgmSrvr] INFO     -- Mgmt server state: nodeid 11 freed, m_reserved_nodes 0000000000000662.
2006-01-10 14:24:42 [MgmSrvr] INFO     -- Mgmt server state: nodeid 11 reserved for ip 172.30.16.1, m_reserved_nodes 0000000000000e62.
2006-01-10 14:24:42 [MgmSrvr] INFO     -- Node 3: Communication to Node 11 opened
2006-01-10 14:24:42 [MgmSrvr] INFO     -- Node 3: Node 11 Connected
2006-01-10 14:24:42 [MgmSrvr] INFO     -- Node 3: Node 11: API version 5.1.2
2006-01-10 14:24:43 [MgmSrvr] ALERT    -- Node 3: Node 11 Disconnected
2006-01-10 14:24:43 [MgmSrvr] INFO     -- Node 3: Communication to Node 11 closed
2006-01-10 14:24:44 [MgmSrvr] INFO     -- Mgmt server state: nodeid 11 freed, m_reserved_nodes 0000000000000662.
2006-01-10 14:24:45 [MgmSrvr] INFO     -- Mgmt server state: nodeid 11 reserved for ip 172.30.16.1, m_reserved_nodes 0000000000000e62.
2006-01-10 14:24:46 [MgmSrvr] INFO     -- Node 3: Communication to Node 11 opened
2006-01-10 14:24:47 [MgmSrvr] INFO     -- Node 3: Node 11 Connected
2006-01-10 14:24:47 [MgmSrvr] INFO     -- Node 3: Node 11: API version 5.1.2
2006-01-10 14:24:47 [MgmSrvr] ALERT    -- Node 3: Node 11 Disconnected
2006-01-10 14:24:47 [MgmSrvr] INFO     -- Node 3: Communication to Node 11 closed
2006-01-10 14:24:48 [MgmSrvr] INFO     -- Mgmt server state: nodeid 11 freed, m_reserved_nodes 0000000000000662.
2006-01-10 14:24:50 [MgmSrvr] INFO     -- Node 3: Communication to Node 11 opened
2006-01-10 14:25:45 [MgmSrvr] INFO     -- Node 3: Backup 10 started from node 1 completed
 StartGCP: 172493 StopGCP: 172626
 #Records: 5342128 #LogRecords: 1428
 Data: 768994140 bytes Log: 57416 bytes
2006-01-10 14:25:46 [MgmSrvr] INFO     -- Node 4: Start phase 5 completed (system restart)
2006-01-10 14:25:46 [MgmSrvr] INFO     -- Node 4: Start phase 6 completed (system restart)
2006-01-10 14:25:46 [MgmSrvr] INFO     -- Node 4: President restarts arbitration thread [state=1]
2006-01-10 14:25:46 [MgmSrvr] INFO     -- Node 4: Start phase 7 completed (system restart)

How to repeat:
see above

http://lists.mysql.com/commits/4321

A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/4359

pushed into 5.1.10

pushed into 5.0.21

pushed into 4.1.19

Thank you for your bug report. This issue has been committed to our
source repository of that product and will be incorporated into the
next release.

If necessary, you can access the source repository and build the latest
available version, including the bugfix, yourself. More information 
about accessing the source trees is available at
    http://www.mysql.com/doc/en/Installing_source_tree.html

Additional info:

Documented bugfix in 4.1.19, 5.0.21, and 5.1.10 changelogs. Closed.