MySQL Bugs: #113428: Invalid signature when setting up mysql-apt-config (0.8.28-1)

Bug #113428	Invalid signature when setting up mysql-apt-config (0.8.28-1)
Submitted:	14 Dec 2023 20:28	Modified:	15 Dec 2023 17:34
Reporter:	Jordan Lantz	Email Updates:
Status:	Closed	Impact on me:	None
Category:	MySQL Package Repos	Severity:	S2 (Serious)
Version:	0.8.28-1	OS:	Debian (bullseye)
Assigned to:		CPU Architecture:	x86 (linux/amd64)

Description:
Started taking this error in our build process today.  Our last build execution was two days ago and was successful.

FROM ruby:3.1.2-slim-bullseye

RUN <<eot
  apt-get -qq update
  apt-get install -y -qq wget
  wget -O /tmp/mysql.deb https://dev.mysql.com/get/mysql-apt-config_0.8.28-1_all.deb
  dpkg -i /tmp/mysql.deb
  apt-get -qq update
  apt-get install -qq -y libmysqlclient-dev mysql-client
eot

output: 
#14 9.613 Preparing to unpack /tmp/mysql.deb ...
#14 9.617 Unpacking mysql-apt-config (0.8.28-1) ...
#14 9.639 Setting up mysql-apt-config (0.8.28-1) ...
#14 11.13 W: GPG error: http://repo.mysql.com/apt/debian bullseye InRelease: The following signatures were invalid: EXPKEYSIG 467B942D3A79BD29 MySQL Release Engineering <mysql-build@oss.oracle.com>
#14 11.13 E: The repository 'http://repo.mysql.com/apt/debian bullseye InRelease' is not signed.
#14 11.79 E: Package 'libmysqlclient-dev' has no installation candidate
#14 11.79 E: Package 'mysql-client' has no installation candidate

How to repeat:
Follow dpkg and apt-get commands listed in description

This is a critical/urgent issue for us -- breaking our production builds and disrupting dev environments team-wide. Please fix ASAP!

We're running into the same problem. I can recreate it on a fresh Ubuntu server using the instructions from the documentation:

apt-key adv --keyserver pgp.mit.edu --recv-keys 3A79BD29
wget https://dev.mysql.com/get/mysql-apt-config_0.8.28-1_all.deb
DEBIAN_FRONTEND=noninteractive dpkg -i mysql-apt-config_0.8.28-1_all.deb
apt-get update
...
Err:5 http://repo.mysql.com/apt/ubuntu lunar InRelease
  The following signatures were invalid: EXPKEYSIG 467B942D3A79BD29 MySQL Release Engineering <mysql-build@oss.oracle.com>
Reading package lists... Done
W: GPG error: http://repo.mysql.com/apt/ubuntu lunar InRelease: The following signatures were invalid: EXPKEYSIG 467B942D3A79BD29 MySQL Release Engineering <mysql-build@oss.oracle.com>
E: The repository 'http://repo.mysql.com/apt/ubuntu lunar InRelease' is not signed.
N: Updating from such a repository can't be done securely, and is therefore disabled by default.
N: See apt-secure(8) manpage for repository creation and user configuration details.

This is also blocking our production and dev deployments. First occurrence we noticed was 16:05 UTC.

It's worth noting that the new keys as specified in the documentation do NOT seem to work.
https://dev.mysql.com/doc/refman/8.0/en/checking-gpg-signature.html

This is a huge blocker for us, all builds are failing and deployment is not possible.

https://repo.mysql.com/apt/ubuntu/conf/distributions is what will change when this is fixed.  I am vivaciously hitting refresh on https://repo.mysql.com/apt/ubuntu/conf to see the mtime change.

Hello Jordan Lantz,

Thank you for the report and feedback.

regards,
Umesh

It also affects all Airflow user. This is a serious problem for anyone who installs MySQL repo because the old key that the repo is signed with is expired.

We are seriously considering switching to MariaDB and suggest it to our users.

https://github.com/apache/airflow/issues/36231

Hi,

thanks for your feedback. We are working on a fix.

Moved to mysql-apt-config_0.8.29-1 and that has resolved the issue.  thank you

Thanks for the bug report. We have rebuilt mysql-apt-config (mysql-apt-config_0.8.29-1_all.deb) with the latest GPG Key, refreshed the repo metadata and published them.

Thanks for the fix. I think however the policy of Oracle/MySQL to have expiry date for your software is deeply flawed.

We had to manually fix all ~50 images we released in the past of our for Apache Airflow because of the expiry date.

Nobody else does it. Postgres, MariaDB, even MsSQL put no expiry date on the keys that are used to sign repos.

By putting expiry key on your apt repository you basically put an expiry date on your software and this expiry date gets shorter and shorter.

A good example of that are your own images that are affected. We had a user asking us for help in Airflow repo https://github.com/apache/airflow/issues/36231#issuecomment-1858419966
`
to help to fix the same issue with `mysql:8.0.35-debian` image of yours and we sent them to your support (as well, you should deal with your own problems).

This image has been released just 25 days ago. And due to the flawed policy of having an expiry date on your key, effectively lifetime of this image was 24 days. Not much. And likely you have a number of those images (similarly as what we had 50 of ours). Now I guess you need to retroactively rebuild/patch your images - which is something the flawed policy of yours made us to get 36 hours of scrambling and and answering support issues of our users (which we did despite our team is made of volunteers, not paid staff as is in the case of MySQL/Oracle).

We kinda lost faith in Oracle being a good steward of MySQL apt repos and we decided in Apache Airflow in accelerated discussion and (currently running) lazy consensus, to switch to MariaDB clients for all our future releases (including the 2.8.0 release that was actually delayed by at least 2 days because of this bug).

Lazy consensus thread here: https://lists.apache.org/list.html?dev@airflow.apache.org

I hope - for the sake of your users loosing days due to such issues, you will reconsider your policies around signing your APT repos.