MySQL Bugs: #104660: Api nodes are showing inconsistent data on K8s

Bug #104660	Api nodes are showing inconsistent data on K8s
Submitted:	18 Aug 2021 23:06	Modified:	25 Aug 2021 11:56
Reporter:	Sainath Reddy Vadyala	Email Updates:
Status:	Not a Bug	Impact on me:	None
Category:	MySQL Server	Severity:	S3 (Non-critical)
Version:	8.0.23	OS:	Debian
Assigned to:	MySQL Verification Team	CPU Architecture:	x86

Description:
2 Manager nodes and 4  Data nodes are on GCP VM's
and 4 Mysql server nodes are on Kubernetes, with PVCs.

The Api nodes are showing data inconsistency when querying the same data on 2 different servers.
Out of 4 only 1 is showing latest data and 3 others are showing no data.

How to repeat:

The Api nodes are showing data inconsistency when querying the same data on 2 different servers.
Out of 4 only 1 is showing latest data and 3 others are showing no data.

Hi,

I'm not sure what are you talking about. Is this about MySQL Cluster (NDBCLUSTER) as that one has API nodes and DATA nodes? Can you share your config? 
What do you mean by "api nodes showing inconsistent data"?

Thanks
Bogdan

I am sorry for the incomplete details. Yes it is a MYSQL Cluster (NDB). Here is my configuration.

2 Manager nodes (GCP VM instances)
4 Data Nodes (GCP VM instances)
4 Sql nodes (Kubernetes pods)

When running a same query on the Sql nodes, One node is showing the correct data but other 3 nodes were showing wrong data.

Last week I found that the Engine on correct data node is NDB Cluster but the other 3 were showing MyISAM.
Now we know the problem but trying to understand why the engine was showing wrong on other 3 nodes, Initially when the table was created the engine was specified as NDB Cluster.

For now we are able to fix the issue by reimporting the table but this seems to be happening very frequently.

I will attach the configuration details in a text file.

Hi,

> When running a same query on the Sql nodes, 
> One node is showing the correct data but other 
> 3 nodes were showing wrong data.

It is physically not possible to have this if all SQL nodes are connected to ndbcluster. The only way you can have different data is if you have MyISAM or InnoDB tables on some of the nodes. If you are properly connected to the ndbcluster engine there is no way you can read different data.

> Last week I found that the Engine on correct data node is NDB Cluster but the other 3 were showing MyISAM.

There is no way MySQL can replace ndbcluster table with MyISAM table and even if that was true there is no way MySQL can invent data to populate that MyISAM table, so the bug is on your side in your application. Something is creating and populating that table.

Thanks
Bogdan