Description:
Hi..!!
Repeatedly facing the problem in MySQL TDE Encryption on tables, deployed in innodb cluster on Kubernetes. Pods are continuously getting into the CrashLoopBackOff state due to following Error and could not be recovered normally.
mysql> SELECT PLUGIN_NAME, PLUGIN_STATUS
-> FROM INFORMATION_SCHEMA.PLUGINS
-> WHERE PLUGIN_NAME LIKE 'keyring%';
+--------------+---------------+
| PLUGIN_NAME | PLUGIN_STATUS |
+--------------+---------------+
| keyring_file | ACTIVE |
| keyring_udf | ACTIVE |
+--------------+---------------+
2 rows in set (0.0020 sec)
show variables like '%keyring%' ;
+--------------------+------------------------------------+
| Variable_name | Value |
+--------------------+------------------------------------+
| keyring_file_data | /mysql_datadir/mysql_files/keyring |
| keyring_operations | ON |
+--------------------+------------------------------------+
2 rows in set (0.0026 sec)
[ERROR] [MY-012226] [InnoDB] Encryption can't find master key, please check the keyring plugin is loaded.
[ERROR] [MY-012179] [InnoDB] Could not find any file associated with the tablespace ID: 690
kubectl get pods |grep fs-mysql
db-mysqlinnocluster-node-0 0/1 CrashLoopBackOff 164 23d
db-mysqlinnocluster-node-1 0/1 CrashLoopBackOff 165 23d
db-mysqlinnocluster-node-2 1/1 Running 1 23d
2021-06-12T15:23:46.586973-00:00 1 [Note] [MY-012550] [InnoDB] Doing recovery: scanned up to log sequence number 5268639149
2021-06-12T15:23:46.608347-00:00 1 [Note] [MY-013083] [InnoDB] Log background threads are being started...
2021-06-12T15:23:46.608948-00:00 1 [Note] [MY-012532] [InnoDB] Applying a batch of 46 redo log records ...
2021-06-12T15:23:46.613845-00:00 1 [ERROR] [MY-012657] [InnoDB] Encryption can't find master key, please check the keyring plugin is loaded.
2021-06-12T15:23:46.614110-00:00 1 [ERROR] [MY-012226] [InnoDB] Encryption information in datafile: ./PLAN_BUDGET/QRTZ_SCHEDULER_STATE.ibd can't be decrypted, please confirm the keyfile is match and keyring plugin is loaded.
2021-06-12T15:23:46.627782-00:00 1 [Note] [MY-012533] [InnoDB] 100%
2021-06-12T15:23:47.128258-00:00 1 [Note] [MY-012535] [InnoDB] Apply batch completed!
2021-06-12T15:23:47.128701-00:00 1 [Warning] [MY-013541] [InnoDB] Doublewrite page 15 for {space: 690, page_no:4} could not be restored. File name unknown for tablespace ID 690
2021-06-12T15:23:47.128994-00:00 1 [ERROR] [MY-012179] [InnoDB] Could not find any file associated with the tablespace ID: 690
2021-06-12T15:23:47.129175-00:00 1 [ERROR] [MY-012964] [InnoDB] Use --innodb-directories to find the tablespace files. If that fails then use --innodb-force-recovery=1 to ignore this and to permanently lose all changes to the missing tablespace(s)
2021-06-12T15:23:47.229801-00:00 1 [ERROR] [MY-012930] [InnoDB] Plugin initialization aborted with error Generic error.
2021-06-12T15:23:47.628807-00:00 1 [ERROR] [MY-010334] [Server] Failed to initialize DD Storage Engine
2021-06-12T15:23:47.629855-00:00 0 [ERROR] [MY-010020] [Server] Data Dictionary initialization failed.
How to repeat:
when Kubernetes pods are restarting many a times, encrypted data is not being recovered by mysql and pods get continuously into CrashLoopBackOff state, although the key ring is already defined and the key-ring file is already set.
kubectl get pods |grep fs-mysql
db-mysqlinnocluster-node-0 0/1 CrashLoopBackOff 164 23d
db-mysqlinnocluster-node-1 0/1 CrashLoopBackOff 165 23d
db-mysqlinnocluster-node-2 1/1 Running 1 23d
--->> Getting these Errors when I tried to look into the logs of crashing pods.
2021-06-12T15:23:46.586973-00:00 1 [Note] [MY-012550] [InnoDB] Doing recovery: scanned up to log sequence number 5268639149
2021-06-12T15:23:46.608347-00:00 1 [Note] [MY-013083] [InnoDB] Log background threads are being started...
2021-06-12T15:23:46.608948-00:00 1 [Note] [MY-012532] [InnoDB] Applying a batch of 46 redo log records ...
2021-06-12T15:23:46.613845-00:00 1 [ERROR] [MY-012657] [InnoDB] Encryption can't find master key, please check the keyring plugin is loaded.
2021-06-12T15:23:46.614110-00:00 1 [ERROR] [MY-012226] [InnoDB] Encryption information in datafile: ./PLAN_BUDGET/QRTZ_SCHEDULER_STATE.ibd can't be decrypted, please confirm the keyfile is match and keyring plugin is loaded.
2021-06-12T15:23:46.627782-00:00 1 [Note] [MY-012533] [InnoDB] 100%
2021-06-12T15:23:47.128258-00:00 1 [Note] [MY-012535] [InnoDB] Apply batch completed!
2021-06-12T15:23:47.128701-00:00 1 [Warning] [MY-013541] [InnoDB] Doublewrite page 15 for {space: 690, page_no:4} could not be restored. File name unknown for tablespace ID 690
2021-06-12T15:23:47.128994-00:00 1 [ERROR] [MY-012179] [InnoDB] Could not find any file associated with the tablespace ID: 690
2021-06-12T15:23:47.129175-00:00 1 [ERROR] [MY-012964] [InnoDB] Use --innodb-directories to find the tablespace files. If that fails then use --innodb-force-recovery=1 to ignore this and to permanently lose all changes to the missing tablespace(s)
2021-06-12T15:23:47.229801-00:00 1 [ERROR] [MY-012930] [InnoDB] Plugin initialization aborted with error Generic error.
2021-06-12T15:23:47.628807-00:00 1 [ERROR] [MY-010334] [Server] Failed to initialize DD Storage Engine
2021-06-12T15:23:47.629855-00:00 0 [ERROR] [MY-010020] [Server] Data Dictionary initialization failed.
Description: Hi..!! Repeatedly facing the problem in MySQL TDE Encryption on tables, deployed in innodb cluster on Kubernetes. Pods are continuously getting into the CrashLoopBackOff state due to following Error and could not be recovered normally. mysql> SELECT PLUGIN_NAME, PLUGIN_STATUS -> FROM INFORMATION_SCHEMA.PLUGINS -> WHERE PLUGIN_NAME LIKE 'keyring%'; +--------------+---------------+ | PLUGIN_NAME | PLUGIN_STATUS | +--------------+---------------+ | keyring_file | ACTIVE | | keyring_udf | ACTIVE | +--------------+---------------+ 2 rows in set (0.0020 sec) show variables like '%keyring%' ; +--------------------+------------------------------------+ | Variable_name | Value | +--------------------+------------------------------------+ | keyring_file_data | /mysql_datadir/mysql_files/keyring | | keyring_operations | ON | +--------------------+------------------------------------+ 2 rows in set (0.0026 sec) [ERROR] [MY-012226] [InnoDB] Encryption can't find master key, please check the keyring plugin is loaded. [ERROR] [MY-012179] [InnoDB] Could not find any file associated with the tablespace ID: 690 kubectl get pods |grep fs-mysql db-mysqlinnocluster-node-0 0/1 CrashLoopBackOff 164 23d db-mysqlinnocluster-node-1 0/1 CrashLoopBackOff 165 23d db-mysqlinnocluster-node-2 1/1 Running 1 23d 2021-06-12T15:23:46.586973-00:00 1 [Note] [MY-012550] [InnoDB] Doing recovery: scanned up to log sequence number 5268639149 2021-06-12T15:23:46.608347-00:00 1 [Note] [MY-013083] [InnoDB] Log background threads are being started... 2021-06-12T15:23:46.608948-00:00 1 [Note] [MY-012532] [InnoDB] Applying a batch of 46 redo log records ... 2021-06-12T15:23:46.613845-00:00 1 [ERROR] [MY-012657] [InnoDB] Encryption can't find master key, please check the keyring plugin is loaded. 2021-06-12T15:23:46.614110-00:00 1 [ERROR] [MY-012226] [InnoDB] Encryption information in datafile: ./PLAN_BUDGET/QRTZ_SCHEDULER_STATE.ibd can't be decrypted, please confirm the keyfile is match and keyring plugin is loaded. 2021-06-12T15:23:46.627782-00:00 1 [Note] [MY-012533] [InnoDB] 100% 2021-06-12T15:23:47.128258-00:00 1 [Note] [MY-012535] [InnoDB] Apply batch completed! 2021-06-12T15:23:47.128701-00:00 1 [Warning] [MY-013541] [InnoDB] Doublewrite page 15 for {space: 690, page_no:4} could not be restored. File name unknown for tablespace ID 690 2021-06-12T15:23:47.128994-00:00 1 [ERROR] [MY-012179] [InnoDB] Could not find any file associated with the tablespace ID: 690 2021-06-12T15:23:47.129175-00:00 1 [ERROR] [MY-012964] [InnoDB] Use --innodb-directories to find the tablespace files. If that fails then use --innodb-force-recovery=1 to ignore this and to permanently lose all changes to the missing tablespace(s) 2021-06-12T15:23:47.229801-00:00 1 [ERROR] [MY-012930] [InnoDB] Plugin initialization aborted with error Generic error. 2021-06-12T15:23:47.628807-00:00 1 [ERROR] [MY-010334] [Server] Failed to initialize DD Storage Engine 2021-06-12T15:23:47.629855-00:00 0 [ERROR] [MY-010020] [Server] Data Dictionary initialization failed. How to repeat: when Kubernetes pods are restarting many a times, encrypted data is not being recovered by mysql and pods get continuously into CrashLoopBackOff state, although the key ring is already defined and the key-ring file is already set. kubectl get pods |grep fs-mysql db-mysqlinnocluster-node-0 0/1 CrashLoopBackOff 164 23d db-mysqlinnocluster-node-1 0/1 CrashLoopBackOff 165 23d db-mysqlinnocluster-node-2 1/1 Running 1 23d --->> Getting these Errors when I tried to look into the logs of crashing pods. 2021-06-12T15:23:46.586973-00:00 1 [Note] [MY-012550] [InnoDB] Doing recovery: scanned up to log sequence number 5268639149 2021-06-12T15:23:46.608347-00:00 1 [Note] [MY-013083] [InnoDB] Log background threads are being started... 2021-06-12T15:23:46.608948-00:00 1 [Note] [MY-012532] [InnoDB] Applying a batch of 46 redo log records ... 2021-06-12T15:23:46.613845-00:00 1 [ERROR] [MY-012657] [InnoDB] Encryption can't find master key, please check the keyring plugin is loaded. 2021-06-12T15:23:46.614110-00:00 1 [ERROR] [MY-012226] [InnoDB] Encryption information in datafile: ./PLAN_BUDGET/QRTZ_SCHEDULER_STATE.ibd can't be decrypted, please confirm the keyfile is match and keyring plugin is loaded. 2021-06-12T15:23:46.627782-00:00 1 [Note] [MY-012533] [InnoDB] 100% 2021-06-12T15:23:47.128258-00:00 1 [Note] [MY-012535] [InnoDB] Apply batch completed! 2021-06-12T15:23:47.128701-00:00 1 [Warning] [MY-013541] [InnoDB] Doublewrite page 15 for {space: 690, page_no:4} could not be restored. File name unknown for tablespace ID 690 2021-06-12T15:23:47.128994-00:00 1 [ERROR] [MY-012179] [InnoDB] Could not find any file associated with the tablespace ID: 690 2021-06-12T15:23:47.129175-00:00 1 [ERROR] [MY-012964] [InnoDB] Use --innodb-directories to find the tablespace files. If that fails then use --innodb-force-recovery=1 to ignore this and to permanently lose all changes to the missing tablespace(s) 2021-06-12T15:23:47.229801-00:00 1 [ERROR] [MY-012930] [InnoDB] Plugin initialization aborted with error Generic error. 2021-06-12T15:23:47.628807-00:00 1 [ERROR] [MY-010334] [Server] Failed to initialize DD Storage Engine 2021-06-12T15:23:47.629855-00:00 0 [ERROR] [MY-010020] [Server] Data Dictionary initialization failed.