MySQL Bugs: #94057: EMPTY cluster-metadata-servers in state.json

Bug #94057	EMPTY cluster-metadata-servers in state.json
Submitted:	25 Jan 2019 10:22	Modified:	8 Mar 2019 22:42
Reporter:	Ivan Ma	Email Updates:
Status:	Closed	Impact on me:	None
Category:	MySQL Router	Severity:	S3 (Non-critical)
Version:	8.0.14	OS:	Any
Assigned to:		CPU Architecture:	Any

Description:
The state.json may contain EMPTY metaserver

{
    "metadata-cache": {
        "group-replication-id": "641d3645-1e10-11e9-8166-0800271b198a",
        "cluster-metadata-servers": []
    },
    "version": "1.0.0"
}

So the RESTART of the Server (start.sh) from the configuration folder will get ERROR

 metadata_cache ERROR [7fe9636ef700] Failed fetching metadata from any of the 0 metadata servers.

Why EMPTY -

1. If we have shutdown (or kill) ALL servers while the ROUTER is still running, the end-result is the ROUTER will cache the last known servers as EMPTY.

2. The network between the ROUTER is failed and it cannot reach any of the servers.

How to repeat:
With a Running InnoDB Cluster + MySQL Router  (assuming on the same servers)

1. kill all servers at once.  pkill -9 mysqld
2. The MySQL Router will see none servers for connectivity.  And that will create EMPTY metaserver entry.

And restart the ROUTER will fail

Suggested fix:
1. The configuration should always contain the STATIC server list which should be used for startup.   

2. EMPTY server list should not be cached.  Server List should only be marked dead and should not be removed.  Possibly to build ACTIVE and INACTIVE LIST.

Hello Ivan,

Thank you for the report.

regards,
Umesh

Posted by developer:
 
Fixed as of the upcoming MySQL Router 8.0.16 release, and here's the changelog entry:

The cached metadata information (state.json) would cache an empty list if
all MySQL servers were shut down; and this caused restarting MySQL Router
to fail. An empty list is no longer cached.

Thank you for the bug report.

Bug #95672 is marked as duplicate of this one