Bug #96174 Certain CDN servers don't hold up-to-date content for https://repo.mysql.com
Submitted: 11 Jul 2019 16:33 Modified: 23 Jul 2019 13:49
Reporter: iBug Fu Email Updates:
Status: Duplicate Impact on me:
None 
Category:MySQL Package Repos and Docker Images Severity:S3 (Non-critical)
Version: OS:Any
Assigned to: MySQL Verification Team CPU Architecture:Any

[11 Jul 2019 16:33] iBug Fu
Description:
In certain networks, https://repo.mysql.com holds outdated content for end users. Terminal log attached below:

ubuntu@iBug-Server:~$ curl -fsSL https://repo.mysql.com/yum/mysql-5.7-community/el/7/x86_64/ | grep -F 5.7.26
1|ubuntu@iBug-Server:~$ curl -fsSL http://repo.mysql.com/yum/mysql-5.7-community/el/7/x86_64/ | grep -F 5.7.26
<IMG SRC="/icons/generic.gif" ALT="[FILE]"> <A HREF="mysql-community-client-5.7.26-1.el7.i686.rpm">mysql-community-client-5.7.26-..&gt;</A> 23-Apr-2019 13:39  23.6M
<IMG SRC="/icons/generic.gif" ALT="[FILE]"> <A HREF="mysql-community-client-5.7.26-1.el7.x86_64.rpm">mysql-community-client-5.7.26-..&gt;</A> 23-Apr-2019 13:39  24.2M
<IMG SRC="/icons/generic.gif" ALT="[FILE]"> <A HREF="mysql-community-common-5.7.26-1.el7.i686.rpm">mysql-community-common-5.7.26-..&gt;</A> 23-Apr-2019 13:39   274k
<IMG SRC="/icons/generic.gif" ALT="[FILE]"> <A HREF="mysql-community-common-5.7.26-1.el7.x86_64.rpm">mysql-community-common-5.7.26-..&gt;</A> 23-Apr-2019 13:39   274k
<IMG SRC="/icons/generic.gif" ALT="[FILE]"> <A HREF="mysql-community-devel-5.7.26-1.el7.i686.rpm">mysql-community-devel-5.7.26-1..&gt;</A> 23-Apr-2019 13:39   3.6M
<IMG SRC="/icons/generic.gif" ALT="[FILE]"> <A HREF="mysql-community-devel-5.7.26-1.el7.x86_64.rpm">mysql-community-devel-5.7.26-1..&gt;</A> 23-Apr-2019 13:39   3.7M
<IMG SRC="/icons/generic.gif" ALT="[FILE]"> <A HREF="mysql-community-embedded-5.7.26-1.el7.i686.rpm">mysql-community-embedded-5.7.2..&gt;</A> 23-Apr-2019 13:39  43.7M
<IMG SRC="/icons/generic.gif" ALT="[FILE]"> <A HREF="mysql-community-embedded-5.7.26-1.el7.x86_64.rpm">mysql-community-embedded-5.7.2..&gt;</A> 23-Apr-2019 13:39  44.9M
<IMG SRC="/icons/generic.gif" ALT="[FILE]"> <A HREF="mysql-community-embedded-compat-5.7.26-1.el7.i686.rpm">mysql-community-embedded-compa..&gt;</A> 23-Apr-2019 13:39  22.3M
<IMG SRC="/icons/generic.gif" ALT="[FILE]"> <A HREF="mysql-community-embedded-compat-5.7.26-1.el7.x86_64.rpm">mysql-community-embedded-compa..&gt;</A> 23-Apr-2019 13:39  23.0M
<IMG SRC="/icons/generic.gif" ALT="[FILE]"> <A HREF="mysql-community-embedded-devel-5.7.26-1.el7.i686.rpm">mysql-community-embedded-devel..&gt;</A> 23-Apr-2019 13:39   116M
<IMG SRC="/icons/generic.gif" ALT="[FILE]"> <A HREF="mysql-community-embedded-devel-5.7.26-1.el7.x86_64.rpm">mysql-community-embedded-devel..&gt;</A> 23-Apr-2019 13:39   124M
<IMG SRC="/icons/generic.gif" ALT="[FILE]"> <A HREF="mysql-community-libs-5.7.26-1.el7.i686.rpm">mysql-community-libs-5.7.26-1...&gt;</A> 23-Apr-2019 13:39   2.1M
<IMG SRC="/icons/generic.gif" ALT="[FILE]"> <A HREF="mysql-community-libs-5.7.26-1.el7.x86_64.rpm">mysql-community-libs-5.7.26-1...&gt;</A> 23-Apr-2019 13:39   2.2M
<IMG SRC="/icons/generic.gif" ALT="[FILE]"> <A HREF="mysql-community-libs-compat-5.7.26-1.el7.i686.rpm">mysql-community-libs-compat-5...&gt;</A> 23-Apr-2019 13:39   2.0M
<IMG SRC="/icons/generic.gif" ALT="[FILE]"> <A HREF="mysql-community-libs-compat-5.7.26-1.el7.x86_64.rpm">mysql-community-libs-compat-5...&gt;</A> 23-Apr-2019 13:39   2.0M
<IMG SRC="/icons/generic.gif" ALT="[FILE]"> <A HREF="mysql-community-server-5.7.26-1.el7.x86_64.rpm">mysql-community-server-5.7.26-..&gt;</A> 23-Apr-2019 13:39   166M
<IMG SRC="/icons/generic.gif" ALT="[FILE]"> <A HREF="mysql-community-test-5.7.26-1.el7.x86_64.rpm">mysql-community-test-5.7.26-1...&gt;</A> 23-Apr-2019 13:39   117M
ubuntu@iBug-Server:~$ nslookup repo.mysql.com
Server:         127.0.0.53
Address:        127.0.0.53#53

Non-authoritative answer:
repo.mysql.com  canonical name = repo.mysql.com.edgekey.net.
repo.mysql.com.edgekey.net      canonical name = e7528.d.akamaiedge.net.
Name:   e7528.d.akamaiedge.net
Address: 23.210.221.222
ubuntu@iBug-Server:~$ curl -fsSL https://repo.mysql.com/yum/mysql-5.7-community/el/7/x86_64/ --resolve 'repo.mysql.com:443:23.56.163.203' | grep -F 5.7.26
<IMG SRC="/icons/generic.gif" ALT="[FILE]"> <A HREF="mysql-community-client-5.7.26-1.el7.i686.rpm">mysql-community-client-5.7.26-..&gt;</A> 23-Apr-2019 13:39  23.6M
<IMG SRC="/icons/generic.gif" ALT="[FILE]"> <A HREF="mysql-community-client-5.7.26-1.el7.x86_64.rpm">mysql-community-client-5.7.26-..&gt;</A> 23-Apr-2019 13:39  24.2M
<IMG SRC="/icons/generic.gif" ALT="[FILE]"> <A HREF="mysql-community-common-5.7.26-1.el7.i686.rpm">mysql-community-common-5.7.26-..&gt;</A> 23-Apr-2019 13:39   274k
<IMG SRC="/icons/generic.gif" ALT="[FILE]"> <A HREF="mysql-community-common-5.7.26-1.el7.x86_64.rpm">mysql-community-common-5.7.26-..&gt;</A> 23-Apr-2019 13:39   274k
<IMG SRC="/icons/generic.gif" ALT="[FILE]"> <A HREF="mysql-community-devel-5.7.26-1.el7.i686.rpm">mysql-community-devel-5.7.26-1..&gt;</A> 23-Apr-2019 13:39   3.6M
<IMG SRC="/icons/generic.gif" ALT="[FILE]"> <A HREF="mysql-community-devel-5.7.26-1.el7.x86_64.rpm">mysql-community-devel-5.7.26-1..&gt;</A> 23-Apr-2019 13:39   3.7M
<IMG SRC="/icons/generic.gif" ALT="[FILE]"> <A HREF="mysql-community-embedded-5.7.26-1.el7.i686.rpm">mysql-community-embedded-5.7.2..&gt;</A> 23-Apr-2019 13:39  43.7M
<IMG SRC="/icons/generic.gif" ALT="[FILE]"> <A HREF="mysql-community-embedded-5.7.26-1.el7.x86_64.rpm">mysql-community-embedded-5.7.2..&gt;</A> 23-Apr-2019 13:39  44.9M
<IMG SRC="/icons/generic.gif" ALT="[FILE]"> <A HREF="mysql-community-embedded-compat-5.7.26-1.el7.i686.rpm">mysql-community-embedded-compa..&gt;</A> 23-Apr-2019 13:39  22.3M
<IMG SRC="/icons/generic.gif" ALT="[FILE]"> <A HREF="mysql-community-embedded-compat-5.7.26-1.el7.x86_64.rpm">mysql-community-embedded-compa..&gt;</A> 23-Apr-2019 13:39  23.0M
<IMG SRC="/icons/generic.gif" ALT="[FILE]"> <A HREF="mysql-community-embedded-devel-5.7.26-1.el7.i686.rpm">mysql-community-embedded-devel..&gt;</A> 23-Apr-2019 13:39   116M
<IMG SRC="/icons/generic.gif" ALT="[FILE]"> <A HREF="mysql-community-embedded-devel-5.7.26-1.el7.x86_64.rpm">mysql-community-embedded-devel..&gt;</A> 23-Apr-2019 13:39   124M
<IMG SRC="/icons/generic.gif" ALT="[FILE]"> <A HREF="mysql-community-libs-5.7.26-1.el7.i686.rpm">mysql-community-libs-5.7.26-1...&gt;</A> 23-Apr-2019 13:39   2.1M
<IMG SRC="/icons/generic.gif" ALT="[FILE]"> <A HREF="mysql-community-libs-5.7.26-1.el7.x86_64.rpm">mysql-community-libs-5.7.26-1...&gt;</A> 23-Apr-2019 13:39   2.2M
<IMG SRC="/icons/generic.gif" ALT="[FILE]"> <A HREF="mysql-community-libs-compat-5.7.26-1.el7.i686.rpm">mysql-community-libs-compat-5...&gt;</A> 23-Apr-2019 13:39   2.0M
<IMG SRC="/icons/generic.gif" ALT="[FILE]"> <A HREF="mysql-community-libs-compat-5.7.26-1.el7.x86_64.rpm">mysql-community-libs-compat-5...&gt;</A> 23-Apr-2019 13:39   2.0M
<IMG SRC="/icons/generic.gif" ALT="[FILE]"> <A HREF="mysql-community-server-5.7.26-1.el7.x86_64.rpm">mysql-community-server-5.7.26-..&gt;</A> 23-Apr-2019 13:39   166M
<IMG SRC="/icons/generic.gif" ALT="[FILE]"> <A HREF="mysql-community-test-5.7.26-1.el7.x86_64.rpm">mysql-community-test-5.7.26-1...&gt;</A> 23-Apr-2019 13:39   117M
ubuntu@iBug-Server:~$

How to repeat:
Run 

curl -fsSL https://repo.mysql.com/yum/mysql-5.7-community/el/7/x86_64/ --resolve 'repo.mysql.com:443:23.210.221.222'

and look at the response: its content is severely outdated, but it's only reproducible by visiting this very CDN node and only via HTTPS. You'll get up-to-date information if you try visiting this node via HTTP.

I'm sure this is an issue on the server side because I managed to reproduce this on various network environments, including China, Japan, Hong Kong, US East and US West. The result is consistent across different test locations.

Suggested fix:
Maybe flush the CDN cache?
[11 Jul 2019 17:06] iBug Fu
After more thorough testing, it appears like the files are actually existent and are served to users, but they're not listed in the HTTPS page, which is confusing and could potentially lead to other undesired results.
[23 Jul 2019 13:49] MySQL Verification Team
Hi,

This is duplicate of a bug #89386

we have changed the TTL on those edge servers now to 15minutes

thanks