Bug #101212 Cannot use CLONE plugin for restore within Docker container
Submitted: 16 Oct 2020 15:58 Modified: 19 Oct 2020 18:13
Reporter: Matthew Boehm Email Updates:
Status: Closed Impact on me:
None 
Category:MySQL Server: Clone Plugin Severity:S3 (Non-critical)
Version:8.0.20 OS:Any (Docker)
Assigned to: MySQL Verification Team CPU Architecture:Any
Tags: clone plugin, docker, mysql8

[16 Oct 2020 15:58] Matthew Boehm
Description:
I have 2 mysql 8.0.20 nodes, running in their own docker containers, configured with single-primary GR. I want to add a 3rd node. The binary logs of mysql1 and mysql2 have been purged, thus forcing mysql3 to join using the CLONE plugin.

Because docker controls the process of mysqld, the CLONE plugin is unable to restart mysqld and the entire container fails.

How to repeat:
- Start 2 docker containers running MySQL 8.0.20
- Configure both mysql's for GR and have them join
- Install CLONE plugin
- Run some load to generate data
-- ie: sysbench oltp, etc
- Purge binary logs from both members
-- FLUSH BINARY LOGS; PURGE BINARY LOGS TO 'XX';
- Start container 3, install CLONE plugin, configure for GR, START GROUP_REPLIACTION

Observe:
- mysql3 connects to other members
- distributed recovery cannot proceed with binary logs, fallback to CLONE
- CLONE connects to donor and transfers dataset
- CLONE attempts to restart joiner mysqld but cannot

2020-10-16T15:18:52.892553Z 304 [ERROR] [MY-013462] [Server] Clone shutting down server as RESTART failed. Please start server to complete clone operation.
2020-10-16T15:18:52.892711Z 304 [ERROR] [MY-011569] [Repl] Plugin group_replication reported: 'Internal query: CLONE INSTANCE FROM 'gr_dr'@'aa2da7dc0930':3306 IDENTIFIED BY '*****' REQUIRE NO SSL; result in error. Error number: 3707'
2020-10-16T15:18:52.892797Z 303 [ERROR] [MY-013465] [Repl] Plugin group_replication reported: 'There was an issue when cloning from another server: Error number: 3707 Error message: Restart server failed (mysqld is not managed by supervisor process).'
2020-10-16T15:18:52.892866Z 304 [ERROR] [MY-011710] [Repl] Plugin group_replication reported: 'Failed to establish an internal server connection to execute plugin operations. Server error: 3169. Server error message: Session was killed'
2020-10-16T15:18:52.892894Z 304 [ERROR] [MY-011705] [Repl] Plugin group_replication reported: 'Error running internal SQL query: SET GLOBAL super_read_only= 1;. Internal failure.'
2020-10-16T15:18:52.892987Z 303 [ERROR] [MY-013173] [Repl] Plugin group_replication reported: 'The plugin encountered a critical error and will abort: Cannot re-enable the super read only after clone failure.'
2020-10-16T15:18:52.893077Z 303 [System] [MY-013172] [Server] Received SHUTDOWN from user <internal signal component service>. Shutting down mysqld (Version: 8.0.20-11).
2020-10-16T15:18:53.129966Z 0 [Note] [MY-010067] [Server] Giving 6 client threads a chance to die gracefully
2020-10-16T15:18:53.130030Z 0 [Note] [MY-010117] [Server] Shutting down slave threads
2020-10-16T15:18:53.130066Z 0 [Note] [MY-010054] [Server] Event Scheduler: Killing the scheduler thread, thread id 5
2020-10-16T15:18:53.130077Z 0 [Note] [MY-010050] [Server] Event Scheduler: Waiting for the scheduler thread to reply
2020-10-16T15:18:53.130144Z 0 [Note] [MY-010048] [Server] Event Scheduler: Stopped
2020-10-16T15:18:54.151055Z 0 [Note] [MY-011953] [InnoDB] Page cleaner took 9475ms to flush 200 pages
2020-10-16T15:18:55.130273Z 0 [Note] [MY-010118] [Server] Forcefully disconnecting 4 remaining clients
2020-10-16T15:18:55.130320Z 0 [Warning] [MY-010909] [Server] /usr/sbin/mysqld: Forcing close of thread 289  user: 'root'.
...

mysql eventually shuts down and exits the container

Suggested fix:
Unknown. I'm not a docker expert. Is there some way to trigger a container restart from within a container?

Maybe this just needs to be a documented case which is unsupported?
[19 Oct 2020 11:12] MySQL Verification Team
Hi,

I'm not sure if it's supported to reset docker from within so verifying this that our GR and Doc team can update documentation

Thanks for the report
Bogdan
[19 Oct 2020 17:28] Kenny Gryp
Hi Matthew,

This was fixed with the Docker images 8.0.21.

Diff: https://github.com/mysql/mysql-docker/commit/fe80e8b4f09b971254028a155be768ea2a5d048d#diff...
[19 Oct 2020 17:29] Kenny Gryp
Also make sure to use restart-policy=on-failure as documented on https://dev.mysql.com/doc/refman/8.0/en/docker-mysql-getting-started.html#docker-starting-...
[19 Oct 2020 18:13] Matthew Boehm
Got it. Thanks. Closing.