Bug #96281 Can't initialize function 'clone'; Plugin initialization function failed
Submitted: 23 Jul 6:42 Modified: 13 Sep 11:51
Reporter: Simon Mudd (OCA) Email Updates:
Status: Closed Impact on me:
None 
Category:MySQL Server: Clone Plugin Severity:S3 (Non-critical)
Version:8.0.17 OS:CentOS (7)
Assigned to: CPU Architecture:x86
Tags: Clone, install plugin failed

[23 Jul 6:42] Simon Mudd
Description:
I see that the clone plugin is new in 8.0.17 so want to try it. I find a slave I had running 8.0.16 and upgrade to 8.0.17 and then try to install the plugin which fails. (this box is intended to be the clone source). There was a suggestion that the server should be able to write so I set read_only = 0. This doesn't seem to help.

How to repeat:
08:24:45 [myuser@myhost ~]$ sudo service mysqld stop
Redirecting to /bin/systemctl stop mysqld.service
08:26:55 [myuser@myhost ~]$ ls *.rpm
mysql-community-client-8.0.17-1.el7.x86_64.rpm  mysql-community-embedded-compat-8.0.17-1.el7.x86_64.rpm  mysql-community-server-8.0.17-1.el7.x86_64.rpm
mysql-community-common-8.0.17-1.el7.x86_64.rpm  mysql-community-libs-8.0.17-1.el7.x86_64.rpm             mysql-community-test-8.0.17-1.el7.x86_64.rpm
mysql-community-devel-8.0.17-1.el7.x86_64.rpm   mysql-community-libs-compat-8.0.17-1.el7.x86_64.rpm
08:27:01 [myuser@myhost ~]$ rpm -qa | grep mysql-comm
mysql-community-common-8.0.16-1.el7.x86_64
mysql-community-server-8.0.16-1.el7.x86_64
mysql-community-libs-8.0.16-1.el7.x86_64
mysql-community-client-8.0.16-1.el7.x86_64
mysql-community-libs-compat-8.0.16-1.el7.x86_64
08:27:10 [myuser@myhost ~]$ sudo rpm -Uvh mysql-community-{common,libs,libs-compat,server,client}-8.0.17-1.el7.x86_64.rpm
Preparing...                          ################################# [100%]
Updating / installing...
   1:mysql-community-common-8.0.17-1.e################################# [ 10%]
   2:mysql-community-libs-8.0.17-1.el7################################# [ 20%]
   3:mysql-community-client-8.0.17-1.e################################# [ 30%]
   4:mysql-community-server-8.0.17-1.e################################# [ 40%]
   5:mysql-community-libs-compat-8.0.1################################# [ 50%]
Cleaning up / removing...
   6:mysql-community-server-8.0.16-1.e################################# [ 60%]
   7:mysql-community-client-8.0.16-1.e################################# [ 70%]
   8:mysql-community-libs-compat-8.0.1################################# [ 80%]
   9:mysql-community-libs-8.0.16-1.el7################################# [ 90%]
  10:mysql-community-common-8.0.16-1.e################################# [100%]
08:28:37 [myuser@myhost ~]$ sudo service mysqld start
Redirecting to /bin/systemctl start mysqld.service
08:29:00 [myuser@myhost ~]$ show_replication_status # internal script to show replication status
Server:            myhost.example.com (8.0.17) on /mysql/db/data/, MySQL GTID ON: Auto_Position: 1
Uptime:             0:00
Master:            rep_user@master.example.com:3306
State:             Waiting for master to send event
Semi-sync slave:   Enabled
Slave IO Running:  Yes
Slave SQL Running: Yes (10 workers)
GTID Executed:     0f1fc688-f3cf-11e8-9441-d06726b200d0:1-10313714,
                   17e46f1d-f3c9-11e8-b987-246e96bdc668:1-12621695,
                   2318e22d-0d08-11e9-9287-246e96b7deb4:1-19698568,
                   d1141b8b-0d07-11e9-a47a-246e968fc548:1-5280137
Dependency track:  WRITESET / XXHASH64
Replication Delay: 0   [NO REPLICATION DELAY]
BookingMeta Delay: 0   [NO REPLICATION DELAY]
Native Delay:      0.239860s (IM: 0.236507s)
Heartbeat Delay:   server_id 999999999: 0.33   [NO REPLICATION DELAY]
08:29:16 [myuser@myhost ~]$ mysql
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 1288
Server version: 8.0.17 MySQL Community Server - GPL
...
root@myhost [(none)]> set global read_only = 0;   ### this is a slave so make it writeable as suggested to me
Query OK, 0 rows affected (0.00 sec)

root@myhost [(none)]> INSTALL PLUGIN clone SONAME 'mysql_clone.so';
ERROR 1123 (HY000): Can't initialize function 'clone'; Plugin initialization function failed.
root@myhost [(none)]> exit
Bye
08:30:09 [myuser@myhost ~]$ sudo tail -20 /var/log/mysqld.log
2019-07-23T06:30:02.793121Z 1288 [ERROR] [MY-013272] [Clone] Plugin Clone reported: 'Client: PFS table creation failed.'
2019-07-23T06:30:02.793155Z 1288 [ERROR] [MY-010202] [Server] Plugin 'clone' init function returned error.
2019-07-23T06:30:02.793181Z 1288 [Note] [MY-010733] [Server] Shutting down plugin 'clone'

Note: 8.0.17 is working and replicating fine.
This specific cluster has only ever run 8.0.
I have tried on 2 other similar setups with the same results.
I will upload my /etc/my.cnf configuration file.

Suggested fix:
Make it work so I can test the plugin and clone to another server.
[23 Jul 7:44] Simon Mudd
Also to be clear on exact version used:

[root@myhost ~]# rpm -q centos-release
centos-release-7-6.1810.2.el7.centos.x86_64
[root@myhost ~]# rpm -qi mysql-community-server
Name        : mysql-community-server
Version     : 8.0.17
Release     : 1.el7
Architecture: x86_64
Install Date: Tue 23 Jul 2019 08:28:36 AM CEST
...
Size        : 1942521369
Source RPM  : mysql-community-8.0.17-1.el7.src.rpm
Build Date  : Wed 26 Jun 2019 09:45:53 PM CEST
Build Host  : siv28.no.oracle.com
Packager    : MySQL Release Engineering <mysql-build@oss.oracle.com>
[23 Jul 7:47] Umesh Shastry
Hello Simon,

Thank you for the report and feedback.

regards,
Umesh
[23 Jul 7:52] Umesh Shastry
Test results - 8.0.17

Attachment: 96281.results (application/octet-stream, text), 4.22 KiB.

[23 Jul 9:59] Simon Mudd
Umesh: thanks for confirming that you can repeat this.

Secondly there is some confusion on whether this should work on a slave with --read-only configured. Documentation at https://dev.mysql.com/doc/refman/8.0/en/clone-plugin-installation.html makes no reference to this so  I think some clarification is needed as to whether it should work or not and any read-only settings that are expected.

Personally I think that it should be possible to load the plugin on the server to be used as the clone source where --read-only or maybe even --super-read-only is set.

Let's clarify current requirements and then we can figure out if they seem too strict for the sort of use case I would want to use them on.

I was also asked about using this on a master: if the master does not run 8.0.17 that involves some sort of downtime as the master is upgraded and cloning from a master adds more load to a master and is clearly not something I would want to do for a semi-sync or asynchronous replication setup (if it can be avoided).
[23 Jul 17:38] Simon Mudd
Se seems the cause is this:

usert@myhost [(none)]> show global variables like 'sql_require%';
+-------------------------+-------+
| Variable_name           | Value |
+-------------------------+-------+
| sql_require_primary_key | ON    |
+-------------------------+-------+
1 row in set (0.00 sec)

Changing this to 0 allows the plugin to be loaded. I've not checked the PS tables that get created but suspect that they have no primary key. Probably for these types of tables this check should not be enforced as these tables are not replicated anyway.
[23 Jul 17:44] Simon Mudd
Just to say thanks to those at Oracle who poked and found the cause.

A suggestion that comes from this if possible would be to some how make the lower level error get bubbled up and reported during the P_S initialisation so that if there's a failure the actual cause is visible. The generic PFS error is not really helpful.
[25 Jul 5:44] Umesh Shastry
Updated category
[30 Aug 2:08] River wu
the variables explicit_defaults_for_timestamp = off also effect install fail.
[12 Sep 15:48] Paul Dubois
Posted by developer:
 
Fixed in 8.0.18.

With sql_require_primary_key enabled, clone plugin initialization
failed because it created two dynamic Performance Schema tables,
which do not support indexes or primary keys. The effect of
sql_require_primary_key is not limited to storage engines that can
participate in replication (currently all storage engines except the
Performance Schema)
[13 Sep 11:51] Simon Mudd
Typo?

"The effect of
sql_require_primary_key is now limited to storage engines that can
participate in replication"  perhaps?
[13 Sep 22:42] Paul Dubois
Yes, "now limited" not "not limited". Thanks for spotting that.
[28 Sep 7:09] Umesh Shastry
Bug #96657 marked as duplicate of this one