Bug #102583 Instance not fully removed after doing cluster.removeInstance with wrong case
Submitted: 12 Feb 2021 15:16 Modified: 26 Feb 15:17
Reporter: Keith Lammers Email Updates:
Status: Closed Impact on me:
Category:Shell AdminAPI InnoDB Cluster / ReplicaSet Severity:S3 (Non-critical)
Version:8.0.23 OS:Windows
Assigned to: CPU Architecture:Any

[12 Feb 2021 15:16] Keith Lammers
If you use a lower-case instance name in cluster.removeInstance, the instance will have group replication stopped, but will not be removed from the cluster metadata.

I worked with Miguel Araujo on this, here are some further details:

verbose: 2021-02-12T09:39:54Z: instance01:3306: DELETE FROM mysql_innodb_cluster_metadata.instances WHERE addresses->'$.mysqlClassic' = 'instance02:3306'
This query doesn't do anything, because this is what's registered in the metadata:
addresses: {"mysqlX": "INSTANCE02:33060", "grLocal": "INSTANCE02:33061", "mysqlClassic": "INSTANCE02:3306"}
But then shell issues this query on that instance: STOP GROUP_REPLICATION, so group_replication stops, and the instance leaves the group, but it's not removed from the metadata and that's why it's shown as (MISSING) in .status()

How to repeat:
* Create a 3 node InnoDB Cluster
* Run dba.removeInstance('instance02'), AdminAPI will state that the instance was successfully removed
* Run cluster.status(), the instance is still listed and shown as offline

* If you do cluster.rescan() and choose yes when prompted to remove the missing instance, it will be removed from the metadata
* If you then add it back with cluster.addInstance(), and then do cluster.removeInstance('INSTANCE02'), it will be removed from the cluster metadata successfully

Suggested fix:
Don't allow the user to run dba.removeInstance with a lower-case instance name.
[16 Feb 2021 14:12] MySQL Verification Team
Hello Keith,

Thank you for the report and feedback.
Verified as described.

[26 Feb 15:17] Miguel Araujo
Posted by developer:
Fixed in 8.0.30, as part of the patch to fix BUG#33893435.

Changelog entry added for MySQL Shell 8.0.30:
InnoDB Cluster commands validate that the target instance belongs to the 
cluster before performing the operation. When the server’s address was 
compared against the server instances registered in the cluster’s metadata, 
host names were compared using case sensitivity, which is contrary to the 
rules in RFC 1034 for domain name comparison. This could lead to a target 
instance not being identified as part of the cluster. Host names are now 
checked using a case insensitive comparison.