Bug #103993 Encryption can't find master key Error in Innodb Cluster Deployed on Kubernetes.
Submitted: 12 Jun 16:57 Modified: 28 Jun 17:17
Reporter: Snehal Bhavsar Email Updates:
Status: Need Feedback Impact on me:
None 
Category:MySQL Server: Security: Encryption Severity:S3 (Non-critical)
Version: 8.0.22 OS:CentOS
Assigned to: MySQL Verification Team CPU Architecture:Any

[12 Jun 16:57] Snehal Bhavsar
Description:
Hi..!!

Repeatedly facing the problem in MySQL TDE Encryption on tables, deployed in innodb cluster on Kubernetes. Pods are continuously getting into the CrashLoopBackOff state due to following Error and could not be recovered normally.

mysql> SELECT PLUGIN_NAME, PLUGIN_STATUS
-> FROM INFORMATION_SCHEMA.PLUGINS
-> WHERE PLUGIN_NAME LIKE 'keyring%';
+--------------+---------------+
| PLUGIN_NAME  | PLUGIN_STATUS |
+--------------+---------------+
| keyring_file | ACTIVE        |
| keyring_udf  | ACTIVE        |
+--------------+---------------+
2 rows in set (0.0020 sec)

show variables like '%keyring%' ;
+--------------------+------------------------------------+
| Variable_name      | Value                              |
+--------------------+------------------------------------+
| keyring_file_data  | /mysql_datadir/mysql_files/keyring |
| keyring_operations | ON                                 |
+--------------------+------------------------------------+
2 rows in set (0.0026 sec)

[ERROR] [MY-012226] [InnoDB] Encryption can't find master key, please check the keyring plugin is loaded. 
[ERROR] [MY-012179] [InnoDB] Could not find any file associated with the tablespace ID: 690

kubectl get pods |grep fs-mysql
db-mysqlinnocluster-node-0                                        0/1     CrashLoopBackOff   164        23d
db-mysqlinnocluster-node-1                                        0/1     CrashLoopBackOff   165        23d
db-mysqlinnocluster-node-2                                        1/1     Running            1          23d

2021-06-12T15:23:46.586973-00:00 1 [Note] [MY-012550] [InnoDB] Doing recovery: scanned up to log sequence number 5268639149
2021-06-12T15:23:46.608347-00:00 1 [Note] [MY-013083] [InnoDB] Log background threads are being started...
2021-06-12T15:23:46.608948-00:00 1 [Note] [MY-012532] [InnoDB] Applying a batch of 46 redo log records ...
2021-06-12T15:23:46.613845-00:00 1 [ERROR] [MY-012657] [InnoDB] Encryption can't find master key, please check the keyring plugin is loaded.
2021-06-12T15:23:46.614110-00:00 1 [ERROR] [MY-012226] [InnoDB] Encryption information in datafile: ./PLAN_BUDGET/QRTZ_SCHEDULER_STATE.ibd can't be decrypted, please confirm the keyfile is match and keyring plugin is loaded.
2021-06-12T15:23:46.627782-00:00 1 [Note] [MY-012533] [InnoDB] 100%
2021-06-12T15:23:47.128258-00:00 1 [Note] [MY-012535] [InnoDB] Apply batch completed!
2021-06-12T15:23:47.128701-00:00 1 [Warning] [MY-013541] [InnoDB] Doublewrite page 15 for {space: 690, page_no:4} could not be restored. File name unknown for tablespace ID 690
2021-06-12T15:23:47.128994-00:00 1 [ERROR] [MY-012179] [InnoDB] Could not find any file associated with the tablespace ID: 690
2021-06-12T15:23:47.129175-00:00 1 [ERROR] [MY-012964] [InnoDB] Use --innodb-directories to find the tablespace files. If that fails then use --innodb-force-recovery=1 to ignore this and to permanently lose all changes to the missing tablespace(s)
2021-06-12T15:23:47.229801-00:00 1 [ERROR] [MY-012930] [InnoDB] Plugin initialization aborted with error Generic error.
2021-06-12T15:23:47.628807-00:00 1 [ERROR] [MY-010334] [Server] Failed to initialize DD Storage Engine
2021-06-12T15:23:47.629855-00:00 0 [ERROR] [MY-010020] [Server] Data Dictionary initialization failed.

How to repeat:
when Kubernetes pods are restarting many a times, encrypted data is not being recovered by mysql and pods get continuously into CrashLoopBackOff state, although the key ring is already defined and the key-ring file is already set.

kubectl get pods |grep fs-mysql
db-mysqlinnocluster-node-0                                        0/1     CrashLoopBackOff   164        23d
db-mysqlinnocluster-node-1                                        0/1     CrashLoopBackOff   165        23d
db-mysqlinnocluster-node-2                                        1/1     Running            1          23d

--->> Getting these Errors when I tried to look into the logs of crashing pods.

2021-06-12T15:23:46.586973-00:00 1 [Note] [MY-012550] [InnoDB] Doing recovery: scanned up to log sequence number 5268639149
2021-06-12T15:23:46.608347-00:00 1 [Note] [MY-013083] [InnoDB] Log background threads are being started...
2021-06-12T15:23:46.608948-00:00 1 [Note] [MY-012532] [InnoDB] Applying a batch of 46 redo log records ...
2021-06-12T15:23:46.613845-00:00 1 [ERROR] [MY-012657] [InnoDB] Encryption can't find master key, please check the keyring plugin is loaded.
2021-06-12T15:23:46.614110-00:00 1 [ERROR] [MY-012226] [InnoDB] Encryption information in datafile: ./PLAN_BUDGET/QRTZ_SCHEDULER_STATE.ibd can't be decrypted, please confirm the keyfile is match and keyring plugin is loaded.
2021-06-12T15:23:46.627782-00:00 1 [Note] [MY-012533] [InnoDB] 100%
2021-06-12T15:23:47.128258-00:00 1 [Note] [MY-012535] [InnoDB] Apply batch completed!
2021-06-12T15:23:47.128701-00:00 1 [Warning] [MY-013541] [InnoDB] Doublewrite page 15 for {space: 690, page_no:4} could not be restored. File name unknown for tablespace ID 690
2021-06-12T15:23:47.128994-00:00 1 [ERROR] [MY-012179] [InnoDB] Could not find any file associated with the tablespace ID: 690
2021-06-12T15:23:47.129175-00:00 1 [ERROR] [MY-012964] [InnoDB] Use --innodb-directories to find the tablespace files. If that fails then use --innodb-force-recovery=1 to ignore this and to permanently lose all changes to the missing tablespace(s)
2021-06-12T15:23:47.229801-00:00 1 [ERROR] [MY-012930] [InnoDB] Plugin initialization aborted with error Generic error.
2021-06-12T15:23:47.628807-00:00 1 [ERROR] [MY-010334] [Server] Failed to initialize DD Storage Engine
2021-06-12T15:23:47.629855-00:00 0 [ERROR] [MY-010020] [Server] Data Dictionary initialization failed.
[28 Jun 17:17] MySQL Verification Team
Hi,

I'm having issue reproducing this, either I'm doing something wrong or I misunderstood your problem.

Can you clarify the: "when Kubernetes pods are restarting many a times"?

I deploy on kubernetes and everything worked ok. I restarted bunch of them and it still worked ok. 

Do you maybe have a script that's creating and deploying them and reproducing the issue as I'm stuck at "it works for me" status :(

Thanks
Bogdan