Bug #106808 Failed to recreate object <number> during restart, error 21029.
Submitted: 23 Mar 8:31 Modified: 11 May 8:08
Reporter: shigeru YAMAMOTO Email Updates:
Status: Can't repeat Impact on me:
None 
Category:MySQL Cluster: Cluster (NDB) storage engine Severity:S2 (Serious)
Version:7.6.21 OS:CentOS (7.6.1810)
Assigned to: MySQL Verification Team CPU Architecture:x86 (Xeon(R) CPU E5-2676 v3)

[23 Mar 8:31] shigeru YAMAMOTO
Description:
We are running MySQL Cluster which has 2 data nodes, and 1 management node.
And trying to add new 2 data nodes.

I found that all data nodes cannot boot after `ALTER TABLE ... REORGANIZE PARTITION`.
(data node log when booting.)

```
2022-03-11 14:14:05 [ndbd] INFO     -- NDB start phase 1 completed
2022-03-11 14:14:05 [ndbd] INFO     -- Start phase 2 completed
2022-03-11 14:14:05 [ndbd] INFO     -- Phase 2 did more initialisations, master accepted our start, we started REDO log initialisations
2022-03-11 14:14:05 [ndbd] INFO     -- Start NDB start phase 2
2022-03-11 14:14:06 [ndbd] INFO     -- Request permission to start our node from master Starting
2022-03-11 14:14:06 [ndbd] INFO     -- Request permission to start our node from master Completed
2022-03-11 14:14:06 [ndbd] INFO     -- NDB start phase 2 completed
2022-03-11 14:14:06 [ndbd] INFO     -- Start phase 3 completed
2022-03-11 14:14:06 [ndbd] INFO     -- Phase 3 locked the data dictionary, performed local connecti on setups, we  asked for permission to start our node
2022-03-11 14:14:06 [ndbd] INFO     -- Start NDB start phase 3
2022-03-11 14:14:06 [ndbd] INFO     -- NDB start phase 3 completed
2022-03-11 14:14:06 [ndbd] INFO     -- Start phase 4 completed
2022-03-11 14:14:06 [ndbd] INFO     -- Phase 4 continued preparations of the REDO log
2022-03-11 14:14:06 [ndbd] INFO     -- Request copying of distribution and dictionary information f rom master Starting
2022-03-11 14:14:06 [ndbd] INFO     -- LDM(1):Ready to start execute REDO log phase, prepare REDO l og phase completed
2022-03-11 14:14:06 [ndbd] INFO     -- Receive arbitrator node 1 [ticket=486100011521732d]
2022-03-11 14:14:08 [ndbd] INFO     -- Copying of dictionary information from master Starting
Found pending trans (0) - committing
commit alter 93
commit alter 184
commit alter 590
commit alter 688
Found pending trans (742) - aborting
Found pending trans (743) - aborting
Found pending trans (744) - aborting
Found pending trans (745) - aborting
2022-03-11 14:14:08 [ndbd] INFO     -- Restart recreating table with id = 1
2022-03-11 14:14:08 [ndbd] INFO     -- Restart recreating table with id = 741
2022-03-11 14:14:08 [ndbd] INFO     -- Restart recreating table with id = 699
2022-03-11 14:14:08 [ndbd] INFO     -- Restart recreating table with id = 700
2022-03-11 14:14:08 [ndbd] INFO     -- Restart recreating table with id = 701
2022-03-11 14:14:08 [ndbd] INFO     -- Restart recreating table with id = 702
2022-03-11 14:14:08 [ndbd] INFO     -- Restart recreating table with id = 703
2022-03-11 14:14:08 [ndbd] INFO     -- Restart recreating table with id = 704
2022-03-11 14:14:08 [ndbd] INFO     -- Restart recreating table with id = 705
2022-03-11 14:14:08 [ndbd] INFO     -- Restart recreating table with id = 706
2022-03-11 14:14:08 [ndbd] INFO     -- Restart recreating table with id = 707
2022-03-11 14:14:08 [ndbd] INFO     -- Restart recreating table with id = 708
2022-03-11 14:14:08 [ndbd] INFO     -- Restart recreating table with id = 709
2022-03-11 14:14:08 [ndbd] INFO     -- Restart recreating table with id = 710
2022-03-11 14:14:08 [ndbd] INFO     -- Restart recreating table with id = 711
2022-03-11 14:14:08 [ndbd] INFO     -- Restart recreating table with id = 712
error: [ code: 21029 line: 27125 node: 8 count: 1 status: 0 key: 0 name: '' ]
2022-03-11 14:14:08 [ndbd] INFO     -- Failed to recreate object 712 during restart, error 21029.
2022-03-11 14:14:08 [ndbd] INFO     -- DBDICT (Line: 5259) 0x00000002
2022-03-11 14:14:08 [ndbd] INFO     -- Error handler shutting down system
2022-03-11 14:14:08 [ndbd] ALERT    -- Node 8: Forced node shutdown completed. Occured during start phase 5. Caused by error 2355: 'Failure to restore schema(Resource configuration error). Permanent error, external action needed'.
```

It seems to wrong with table id=712.
Whereas, I restored all data nodes and use  `ALTER TABLE ... ALGORITHM=COPY, ORGANIZE PARTITION`, I can restart data nodes correctly.

How to repeat:
add new data nodes.

reference:
- https://dev.mysql.com/doc/refman/5.7/en/mysql-cluster-online-add-node-example.html

1. setup MySQL Cluster with 2 data nodes
2. add 2 data nodes
3. create new nodegroup
4. reorganize partion
5. stop a data node (id=8)
6. start a data node (id=8)
[23 Mar 8:54] shigeru YAMAMOTO
upload a file (ndb_error_report).

- mysql-bug-106808-ndb_error_report_20220323153507.tar.bz2
[29 Mar 8:11] MySQL Verification Team
Hi,

What **exactly** did you do, as I'm failing to reproduce the problem?

Thanks
[4 Apr 8:07] shigeru YAMAMOTO
I did the following steps:

- mysql-bug-106808-reproduce-steps.txt
[11 May 8:08] shigeru YAMAMOTO
I re-upload file.
It is a step for reproducing the problems.