Bug #94057 EMPTY cluster-metadata-servers in state.json
Submitted: 25 Jan 10:22 Modified: 8 Mar 22:42
Reporter: Ivan Ma Email Updates:
Status: Closed Impact on me:
None 
Category:MySQL Router Severity:S3 (Non-critical)
Version:8.0.14 OS:Any
Assigned to: CPU Architecture:Any

[25 Jan 10:22] Ivan Ma
Description:
The state.json may contain EMPTY metaserver

{
    "metadata-cache": {
        "group-replication-id": "641d3645-1e10-11e9-8166-0800271b198a",
        "cluster-metadata-servers": []
    },
    "version": "1.0.0"
}

So the RESTART of the Server (start.sh) from the configuration folder will get ERROR

 metadata_cache ERROR [7fe9636ef700] Failed fetching metadata from any of the 0 metadata servers.

Why EMPTY -

1. If we have shutdown (or kill) ALL servers while the ROUTER is still running, the end-result is the ROUTER will cache the last known servers as EMPTY.

2. The network between the ROUTER is failed and it cannot reach any of the servers.

How to repeat:
With a Running InnoDB Cluster + MySQL Router  (assuming on the same servers)

1. kill all servers at once.  pkill -9 mysqld
2. The MySQL Router will see none servers for connectivity.  And that will create EMPTY metaserver entry.

And restart the ROUTER will fail

Suggested fix:
1. The configuration should always contain the STATIC server list which should be used for startup.   

2. EMPTY server list should not be cached.  Server List should only be marked dead and should not be removed.  Possibly to build ACTIVE and INACTIVE LIST.
[25 Jan 12:39] Umesh Shastry
Hello Ivan,

Thank you for the report.

regards,
Umesh
[8 Mar 22:42] Philip Olson
Posted by developer:
 
Fixed as of the upcoming MySQL Router 8.0.16 release, and here's the changelog entry:

The cached metadata information (state.json) would cache an empty list if
all MySQL servers were shut down; and this caused restarting MySQL Router
to fail. An empty list is no longer cached.

Thank you for the bug report.
[11 Jun 16:57] Bogdan Kecman
Bug #95672 is marked as duplicate of this one