Description:
I have a 2 slice 2 copy NDB cluster with 1 manager and 4 data nodes. Node1-node5 respectively, where node1 is the manager. Then I noticed after running it for a while that I encountered this error and node5 was forced to shut down:
```
Current byte-offset of file-pointer is: 1067
Time: Saturday 19 October 2024 - 15:06:25
Status: Temporary error, restart node
Message: Node lost connection to other nodes and can not form a unpartitioned cluster, please investigate if there are error(s) on other node(s) (Arbitration error)
Error: 2305
Error data: Arbitrator decided to shutdown this node
Error object: QMGR (Line: 7376) 0x00000002
Program: ndbd
Pid: 2827
Version: mysql-8.0.35 ndb-8.0.35
Trace file name: ndb_5_trace.log.1
Trace file path: /usr/local/mysql-cluster/data/ndb_5_trace.log.1 [t1..t1]
***EOM***
```
How to repeat:
I've summarized the steps of this bug from the log files, and I'll upload all the logs I've collected in the future.
1, 2024-10-19 12:53:55 to 12:53:56: node5 logged multiple table deletion requests (DROP_TAB_REQ), indicating that at this point the cluster is in a stable working state and performing data operations normally.
2. 2024-10-19 15:05:18: node5 logs that node1 disconnected:
```
2024-10-19 15:05:18 [ndbd] INFO -- Node 1 disconnected in state: 0
2024-10-19 15:05:18 [ndbd] INFO -- Lost arbitrator node 1 - process failure [state=6]
```
node1 is the arbitrator node for the cluster. When node1 was disconnected, node5 lost its arbitrator node, causing it to be unable to continue to ensure the proper functioning of the system. The state=6 in the log indicates that node1 experienced a process failure, but I did not find a reason for this occurrence.
3, 2024-10-19 15:05:48: node5 log shows node2 disconnected:
```
2024-10-19 15:05:48 [ndbd] INFO -- Node 2 disconnected in state: 0
```
node2 disconnected. The cluster started triggering node troubleshooting. node5 participated in the troubleshooting of node2 and completed the takeover.
4, 2024-10-19 15:05:59: node5 logged that node3 also disconnected:
```
2024-10-19 15:05:59 [ndbd] INFO -- Node 3 disconnected in state: 0
```
node5 initiated failover processing for node3 again, including taking over the failed node's transactions and resources.
5, 2024-10-19 15:06:25: Arbitration failure and forced shutdown:
```
2024-10-19 15:06:25 [ndbd] ALERT -- Arbitration check lost - missing node group
2024-10-19 15:06:25 [ndbd] ALERT -- Node 5: Forced node shutdown completed. Caused by error 2305: 'Node lost connection to other nodes and can not form a unpartitioned cluster, please investigate if there are error(s) on other node(s)(Arbitration error). Temporary error, restart node'.
```
After dealing with the node failure, node5 attempts to perform an arbitration check, but it is unable to successfully complete the arbitration due to the loss of the arbitration node, node1, and the failure of other nodes such as node2 and node3.
Error 2305 in the log shows that node5 triggered an arbitration error because it could not connect to enough nodes to form an unpartitioned cluster. The system decides to force node5 to shut down.
