Description:
I have 2 mysql 8.0.20 nodes, running in their own docker containers, configured with single-primary GR. I want to add a 3rd node. The binary logs of mysql1 and mysql2 have been purged, thus forcing mysql3 to join using the CLONE plugin.
Because docker controls the process of mysqld, the CLONE plugin is unable to restart mysqld and the entire container fails.
How to repeat:
- Start 2 docker containers running MySQL 8.0.20
- Configure both mysql's for GR and have them join
- Install CLONE plugin
- Run some load to generate data
-- ie: sysbench oltp, etc
- Purge binary logs from both members
-- FLUSH BINARY LOGS; PURGE BINARY LOGS TO 'XX';
- Start container 3, install CLONE plugin, configure for GR, START GROUP_REPLIACTION
Observe:
- mysql3 connects to other members
- distributed recovery cannot proceed with binary logs, fallback to CLONE
- CLONE connects to donor and transfers dataset
- CLONE attempts to restart joiner mysqld but cannot
2020-10-16T15:18:52.892553Z 304 [ERROR] [MY-013462] [Server] Clone shutting down server as RESTART failed. Please start server to complete clone operation.
2020-10-16T15:18:52.892711Z 304 [ERROR] [MY-011569] [Repl] Plugin group_replication reported: 'Internal query: CLONE INSTANCE FROM 'gr_dr'@'aa2da7dc0930':3306 IDENTIFIED BY '*****' REQUIRE NO SSL; result in error. Error number: 3707'
2020-10-16T15:18:52.892797Z 303 [ERROR] [MY-013465] [Repl] Plugin group_replication reported: 'There was an issue when cloning from another server: Error number: 3707 Error message: Restart server failed (mysqld is not managed by supervisor process).'
2020-10-16T15:18:52.892866Z 304 [ERROR] [MY-011710] [Repl] Plugin group_replication reported: 'Failed to establish an internal server connection to execute plugin operations. Server error: 3169. Server error message: Session was killed'
2020-10-16T15:18:52.892894Z 304 [ERROR] [MY-011705] [Repl] Plugin group_replication reported: 'Error running internal SQL query: SET GLOBAL super_read_only= 1;. Internal failure.'
2020-10-16T15:18:52.892987Z 303 [ERROR] [MY-013173] [Repl] Plugin group_replication reported: 'The plugin encountered a critical error and will abort: Cannot re-enable the super read only after clone failure.'
2020-10-16T15:18:52.893077Z 303 [System] [MY-013172] [Server] Received SHUTDOWN from user <internal signal component service>. Shutting down mysqld (Version: 8.0.20-11).
2020-10-16T15:18:53.129966Z 0 [Note] [MY-010067] [Server] Giving 6 client threads a chance to die gracefully
2020-10-16T15:18:53.130030Z 0 [Note] [MY-010117] [Server] Shutting down slave threads
2020-10-16T15:18:53.130066Z 0 [Note] [MY-010054] [Server] Event Scheduler: Killing the scheduler thread, thread id 5
2020-10-16T15:18:53.130077Z 0 [Note] [MY-010050] [Server] Event Scheduler: Waiting for the scheduler thread to reply
2020-10-16T15:18:53.130144Z 0 [Note] [MY-010048] [Server] Event Scheduler: Stopped
2020-10-16T15:18:54.151055Z 0 [Note] [MY-011953] [InnoDB] Page cleaner took 9475ms to flush 200 pages
2020-10-16T15:18:55.130273Z 0 [Note] [MY-010118] [Server] Forcefully disconnecting 4 remaining clients
2020-10-16T15:18:55.130320Z 0 [Warning] [MY-010909] [Server] /usr/sbin/mysqld: Forcing close of thread 289 user: 'root'.
...
mysql eventually shuts down and exits the container
Suggested fix:
Unknown. I'm not a docker expert. Is there some way to trigger a container restart from within a container?
Maybe this just needs to be a documented case which is unsupported?