Bug #99436 NDB fully replicated tables only in node group 0, not in node group 1
Submitted: 4 May 2020 11:51 Modified: 28 Sep 2020 14:46
Reporter: NGUYEN TRUNG HIEU Email Updates:
Status: Not a Bug Impact on me:
None 
Category:MySQL Cluster: Cluster (NDB) storage engine Severity:S3 (Non-critical)
Version:8.0.20 OS:Red Hat (8)
Assigned to: MySQL Verification Team CPU Architecture:Any

[4 May 2020 11:51] NGUYEN TRUNG HIEU
Description:
Fully replicated tables is each data node in each node group has full table records, right?

Ref link:

https://dev.mysql.com/doc/relnotes/mysql-cluster/7.5/en/news-7-5-2.html

Tables using one of the two per-node-group settings for the fragment count type can also be fully replicated. This requires that the table's fragment count type is ONE_PER_NODE_GROUP or ONE_PER_LDM_PER_NODE_GROUP, and can be enabled using the option FULLY_REPLICATED=1 within in an NDB_TABLE comment. The option can be enabled by default for all new NDB tables using the ndb_fully_replicated system variable added in this release.

https://dev.mysql.com/doc/refman/8.0/en/create-table-ndb-table-comment-options.html

1. I have 2 node group: node group 0, node group 1

[root@centos8 ~]# ndb_mgm -e show
Connected to Management Server at: 192.168.1.3:1186
Cluster Configuration
---------------------
[ndbd(NDB)]     4 node(s)
id=1    @192.168.1.5  (mysql-8.0.20 ndb-8.0.20, Nodegroup: 0)
id=2    @192.168.1.6  (mysql-8.0.20 ndb-8.0.20, Nodegroup: 0)
id=3    @192.168.1.11  (mysql-8.0.20 ndb-8.0.20, Nodegroup: 1, *)
id=4    @192.168.1.12  (mysql-8.0.20 ndb-8.0.20, Nodegroup: 1)

[ndb_mgmd(MGM)] 2 node(s)
id=145  @192.168.1.3  (mysql-8.0.20 ndb-8.0.20)
id=146  @192.168.1.4  (mysql-8.0.20 ndb-8.0.20)

[mysqld(API)]   6 node(s)
id=166  @192.168.1.11  (mysql-8.0.20 ndb-8.0.20)
id=167  @192.168.1.11  (mysql-8.0.20 ndb-8.0.20)
id=168 (not connected, accepting connect from any host)
id=169 (not connected, accepting connect from any host)
id=170 (not connected, accepting connect from any host)
id=171 (not connected, accepting connect from any host)

2. I created a NDB table with fully replicated tables

CREATE SCHEMA devops6 DEFAULT CHARACTER SET utf8mb4;

USE devops6;

CREATE TABLE dòngđời
(
	số BIGINT UNSIGNED NOT NULL AUTO_INCREMENT,
	trôiqua VARCHAR(26) NOT NULL,
	PRIMARY KEY (số)
) 
DEFAULT CHARACTER SET = utf8mb4
ENGINE=ndbcluster COMMENT='NDB_TABLE=FULLY_REPLICATED=1 PARTITION_BALANCE=FOR_RP_BY_LDM';

INSERT INTO dòngđời (trôiqua) values (current_timestamp(6));

But fully replicated tables just in node group 0, node in node group 1

[root@centos8 ~]# ndb_desc -c 192.168.1.3:1186 dòngđời -d devops444 -p -n
-- dòngđời --
Version: 16777221
Fragment type: HashMapPartition
K Value: 6
Min load factor: 78
Max load factor: 80
Temporary table: no
Number of attributes: 2
Number of primary keys: 1
Length of frm data: 990
Max Rows: 0
Row Checksum: 1
Row GCI: 1
SingleUserMode: 0
ForceVarPart: 1
PartitionCount: 2
FragmentCount: 4
PartitionBalance: FOR_RP_BY_NODE
ExtraRowGciBits: 0
ExtraRowAuthorBits: 0
TableStatus: Retrieved
Table options: readbackup, fullyreplicated
HashMap: DEFAULT-HASHMAP-3840-2
-- Attributes --
số Bigunsigned PRIMARY KEY DISTRIBUTION KEY AT=FIXED ST=MEMORY AUTO_INCR
trôiqua Varchar(104;utf8mb4_0900_ai_ci) NOT NULL AT=SHORT_VAR ST=MEMORY
-- Indexes --
PRIMARY KEY(số) - UniqueHashIndex
PRIMARY(số) - OrderedIndex
-- Per partition info --
Partition       Row count       Commit count    Frag fixed memory       Frag varsized memory    Extent_space    Free extent_space       Nodes
0               6               6               32768                   32768                   0               0                       1,2
1               4               4               32768                   32768                   0               0      

[root@centos8 ~]# ndb_desc -c 192.168.1.3:1186 dòngđời -d devops555 -p -n
-- dòngđời --
Version: 16777219
Fragment type: HashMapPartition
K Value: 6
Min load factor: 78
Max load factor: 80
Temporary table: no
Number of attributes: 2
Number of primary keys: 1
Length of frm data: 986
Max Rows: 0
Row Checksum: 1
Row GCI: 1
SingleUserMode: 0
ForceVarPart: 1
PartitionCount: 1
FragmentCount: 2
PartitionBalance: FOR_RA_BY_NODE
ExtraRowGciBits: 0
ExtraRowAuthorBits: 0
TableStatus: Retrieved
Table options: readbackup, fullyreplicated
HashMap: DEFAULT-HASHMAP-3840-1
-- Attributes --
số Bigunsigned PRIMARY KEY DISTRIBUTION KEY AT=FIXED ST=MEMORY AUTO_INCR
trôiqua Varchar(104;utf8mb4_0900_ai_ci) NOT NULL AT=SHORT_VAR ST=MEMORY
-- Indexes --
PRIMARY KEY(số) - UniqueHashIndex
PRIMARY(số) - OrderedIndex
-- Per partition info --
Partition       Row count       Commit count    Frag fixed memory       Frag varsized memory    Extent_space    Free extent_space       Nodes
0               11              11              32768                   32768                   0               0                       1,2

[root@centos8 ~]# ndb_desc -c 192.168.1.3:1186 dòngđời -d devops666 -p -n
-- dòngđời --
Version: 16777219
Fragment type: HashMapPartition
K Value: 6
Min load factor: 78
Max load factor: 80
Temporary table: no
Number of attributes: 2
Number of primary keys: 1
Length of frm data: 988
Max Rows: 0
Row Checksum: 1
Row GCI: 1
SingleUserMode: 0
ForceVarPart: 1
PartitionCount: 1
FragmentCount: 2
PartitionBalance: FOR_RA_BY_LDM
ExtraRowGciBits: 0
ExtraRowAuthorBits: 0
TableStatus: Retrieved
Table options: readbackup, fullyreplicated
HashMap: DEFAULT-HASHMAP-3840-1
-- Attributes --
số Bigunsigned PRIMARY KEY DISTRIBUTION KEY AT=FIXED ST=MEMORY AUTO_INCR
trôiqua Varchar(104;utf8mb4_0900_ai_ci) NOT NULL AT=SHORT_VAR ST=MEMORY
-- Indexes --
PRIMARY KEY(số) - UniqueHashIndex
PRIMARY(số) - OrderedIndex
-- Per partition info --
Partition       Row count       Commit count    Frag fixed memory       Frag varsized memory    Extent_space    Free extent_space       Nodes
0               12              12              32768                   32768                   0               0                       1,2

[root@centos8 ~]# ndb_desc -c 192.168.1.3:1186 dòngđời -d devops777 -p -n
-- dòngđời --
Version: 16777218
Fragment type: HashMapPartition
K Value: 6
Min load factor: 78
Max load factor: 80
Temporary table: no
Number of attributes: 2
Number of primary keys: 1
Length of frm data: 989
Max Rows: 0
Row Checksum: 1
Row GCI: 1
SingleUserMode: 0
ForceVarPart: 1
PartitionCount: 2
FragmentCount: 4
PartitionBalance: FOR_RP_BY_LDM
ExtraRowGciBits: 0
ExtraRowAuthorBits: 0
TableStatus: Retrieved
Table options: readbackup, fullyreplicated
HashMap: DEFAULT-HASHMAP-3840-2
-- Attributes --
số Bigunsigned PRIMARY KEY DISTRIBUTION KEY AT=FIXED ST=MEMORY AUTO_INCR
trôiqua Varchar(104;utf8mb4_0900_ai_ci) NOT NULL AT=SHORT_VAR ST=MEMORY
-- Indexes --
PRIMARY KEY(số) - UniqueHashIndex
PRIMARY(số) - OrderedIndex
-- Per partition info --
Partition       Row count       Commit count    Frag fixed memory       Frag varsized memory    Extent_space    Free extent_space       Nodes
0               6               6               32768                   32768                   0               0                       1,2
1               6               6               32768                   32768                   0               0                       2,1

Please help me

Thank you very much!

How to repeat:
--> and FOR_RP_BY_NODE, FOR_RA_BY_NODE, FOR_RA_BY_LDM
[4 May 2020 11:54] NGUYEN TRUNG HIEU
I reorganize partition but it's not effect

mysql> ALTER TABLE devops444.dòngđời ALGORITHM=INPLACE, REORGANIZE PARTITION;
Query OK, 0 rows affected (0.35 sec)
Records: 0  Duplicates: 0  Warnings: 0

mysql> ALTER TABLE devops555.dòngđời ALGORITHM=INPLACE, REORGANIZE PARTITION;
Query OK, 0 rows affected (0.37 sec)
Records: 0  Duplicates: 0  Warnings: 0

mysql> ALTER TABLE devops666.dòngđời ALGORITHM=INPLACE, REORGANIZE PARTITION;
Query OK, 0 rows affected (0.36 sec)
Records: 0  Duplicates: 0  Warnings: 0

mysql> ALTER TABLE devops777.dòngđời ALGORITHM=INPLACE, REORGANIZE PARTITION;
Query OK, 0 rows affected (0.38 sec)
Records: 0  Duplicates: 0  Warnings: 0
[28 Sep 2020 14:43] MySQL Verification Team
Hi,

> Fully replicated tables is each data node in each node group has full table records, right?

No. It depends on the number of data nodes. This is true only for the 2 datanode configuration. When you have more than 2 data nodes this statement is false.
[28 Sep 2020 14:46] MySQL Verification Team
> id=1    @192.168.1.5  (mysql-8.0.20 ndb-8.0.20, Nodegroup: 0)
> id=2    @192.168.1.6  (mysql-8.0.20 ndb-8.0.20, Nodegroup: 0)
> id=3    @192.168.1.11  (mysql-8.0.20 ndb-8.0.20, Nodegroup: 1, *)
> id=4    @192.168.1.12  (mysql-8.0.20 ndb-8.0.20, Nodegroup: 1)

In this case, half of your table is with nodes 1 and 2 and other half of the table is on nodes 3 and 4. That's where data is stored but when all 4 nodes are up 1/4 of the table is accessed from each node (the other 1/4 of data that exist on that node is accessed only if the pair node dies)

The only way to have whole table on all 4 data nodes is to use noofreplica=4

all best
Bogdan