Bug #113428 Invalid signature when setting up mysql-apt-config (0.8.28-1)
Submitted: 14 Dec 2023 20:28 Modified: 15 Dec 2023 17:34
Reporter: Jordan Lantz Email Updates:
Status: Closed Impact on me:
Category:MySQL Package Repos Severity:S2 (Serious)
Version:0.8.28-1 OS:Debian (bullseye)
Assigned to: CPU Architecture:x86 (linux/amd64)

[14 Dec 2023 20:28] Jordan Lantz
Started taking this error in our build process today.  Our last build execution was two days ago and was successful.

FROM ruby:3.1.2-slim-bullseye

RUN <<eot
  apt-get -qq update
  apt-get install -y -qq wget
  wget -O /tmp/mysql.deb https://dev.mysql.com/get/mysql-apt-config_0.8.28-1_all.deb
  dpkg -i /tmp/mysql.deb
  apt-get -qq update
  apt-get install -qq -y libmysqlclient-dev mysql-client

#14 9.613 Preparing to unpack /tmp/mysql.deb ...
#14 9.617 Unpacking mysql-apt-config (0.8.28-1) ...
#14 9.639 Setting up mysql-apt-config (0.8.28-1) ...
#14 11.13 W: GPG error: http://repo.mysql.com/apt/debian bullseye InRelease: The following signatures were invalid: EXPKEYSIG 467B942D3A79BD29 MySQL Release Engineering <mysql-build@oss.oracle.com>
#14 11.13 E: The repository 'http://repo.mysql.com/apt/debian bullseye InRelease' is not signed.
#14 11.79 E: Package 'libmysqlclient-dev' has no installation candidate
#14 11.79 E: Package 'mysql-client' has no installation candidate

How to repeat:
Follow dpkg and apt-get commands listed in description
[14 Dec 2023 21:04] Jordan Smith
This is a critical/urgent issue for us -- breaking our production builds and disrupting dev environments team-wide. Please fix ASAP!
[14 Dec 2023 21:22] Reed McLean
We're running into the same problem. I can recreate it on a fresh Ubuntu server using the instructions from the documentation:

apt-key adv --keyserver pgp.mit.edu --recv-keys 3A79BD29
wget https://dev.mysql.com/get/mysql-apt-config_0.8.28-1_all.deb
DEBIAN_FRONTEND=noninteractive dpkg -i mysql-apt-config_0.8.28-1_all.deb
apt-get update
Err:5 http://repo.mysql.com/apt/ubuntu lunar InRelease
  The following signatures were invalid: EXPKEYSIG 467B942D3A79BD29 MySQL Release Engineering <mysql-build@oss.oracle.com>
Reading package lists... Done
W: GPG error: http://repo.mysql.com/apt/ubuntu lunar InRelease: The following signatures were invalid: EXPKEYSIG 467B942D3A79BD29 MySQL Release Engineering <mysql-build@oss.oracle.com>
E: The repository 'http://repo.mysql.com/apt/ubuntu lunar InRelease' is not signed.
N: Updating from such a repository can't be done securely, and is therefore disabled by default.
N: See apt-secure(8) manpage for repository creation and user configuration details.

This is also blocking our production and dev deployments. First occurrence we noticed was 16:05 UTC.
[14 Dec 2023 22:01] Jen Guerra
It's worth noting that the new keys as specified in the documentation do NOT seem to work.

This is a huge blocker for us, all builds are failing and deployment is not possible.
[14 Dec 2023 23:36] Ben Sherman
https://repo.mysql.com/apt/ubuntu/conf/distributions is what will change when this is fixed.  I am vivaciously hitting refresh on https://repo.mysql.com/apt/ubuntu/conf to see the mtime change.
[15 Dec 2023 5:48] MySQL Verification Team
Hello Jordan Lantz,

Thank you for the report and feedback.

[15 Dec 2023 8:07] Jarosław Potiuk
It also affects all Airflow user. This is a serious problem for anyone who installs MySQL repo because the old key that the repo is signed with is expired.

We are seriously considering switching to MariaDB and suggest it to our users.

[15 Dec 2023 9:25] Terje Røsten

thanks for your feedback. We are working on a fix.
[15 Dec 2023 15:49] Jordan Lantz
Moved to mysql-apt-config_0.8.29-1 and that has resolved the issue.  thank you
[15 Dec 2023 17:34] Balasubramanian Kandasamy
Thanks for the bug report. We have rebuilt mysql-apt-config (mysql-apt-config_0.8.29-1_all.deb) with the latest GPG Key, refreshed the repo metadata and published them.
[16 Dec 2023 4:20] Jarosław Potiuk
Thanks for the fix. I think however the policy of Oracle/MySQL to have expiry date for your software is deeply flawed. 

We had to manually fix all ~50 images we released in the past of our for Apache Airflow because of the expiry date. 

Nobody else does it. Postgres, MariaDB, even MsSQL put no expiry date on the keys that are used to sign repos. 

By putting expiry key on your apt repository you basically put an expiry date on your software and this expiry date gets shorter and shorter.

A good example of that are your own images that are affected. We had a user asking us for help in Airflow repo https://github.com/apache/airflow/issues/36231#issuecomment-1858419966 
to help to fix the same issue with  `mysql:8.0.35-debian` image of yours and we sent them to your support (as well, you should deal with your own problems).

This image has been released just 25 days ago. And due to the flawed policy of having an expiry date on your key, effectively lifetime of this image was 24 days. Not much. And likely you have a number of those images (similarly as what we had 50 of ours).  Now I guess you need to retroactively rebuild/patch your images - which is something the flawed policy of yours made us to get 36 hours of scrambling and  and answering support issues of our users (which we did despite our team is made of volunteers, not paid staff as is in the case of MySQL/Oracle). 

We kinda lost faith in Oracle being a good steward of MySQL apt repos and we decided in Apache Airflow in accelerated discussion and (currently running) lazy consensus, to switch to MariaDB clients for all our future releases (including the 2.8.0 release that was actually delayed by at least 2 days because of this bug).

Lazy consensus thread here: https://lists.apache.org/list.html?dev@airflow.apache.org

I hope - for the sake of your users loosing days due to such issues, you will reconsider your policies around signing your APT repos.