MySQL Bugs: #89386: Index pages of YUM repos is not current

Bug #89386	Index pages of YUM repos is not current
Submitted:	24 Jan 2018 15:39	Modified:	11 Jun 2019 15:07
Reporter:	Daniel Holmes	Email Updates:
Status:	Not a Bug	Impact on me:	None
Category:	MySQL Package Repos	Severity:	S2 (Serious)
Version:		OS:	Red Hat
Assigned to:	MySQL Verification Team	CPU Architecture:	Any

Description:
The generated index files for these repos are not current.  Specifically, this RPM file
repo.mysql.com/yum/mysql-5.7-community/el/7/x86_64/mysql-community-common-5.7.21-1.el7.x86_64.rpm

is posted but not listed on the index page 
repo.mysql.com/yum/mysql-5.7-community/el/7/x86_64/

This is preventing our environment from being able to mirror in a copy of the RPM.  

How to repeat:
Verify the index URL inthe description and see that the mysql-community-common-5.7.21-1.el7.x86_64.rpm is not in the listing.

Suggested fix:
Please regenerate the index files to be accurate.

Bonus: If rsync could be enabled on repo.mysql.com that would be even better so that we could rsync the mirror rather than having to depend on a web crawl method.  This is necessary in our environment because we need to push a mirror to an area of our network that does not have direct internet access available.

Hi!

This is due to limitation in content delivery network (CDN),
I think you can use a tool that understand repo meta data to
keep a local mirror current, search for e.g. reposync & 
createrepo.

Yes, I do know about reposync as a tool.  Problem with that is needing to configure an overall repo file to cover each of the repos in your overall site and then it doesn't pick up new versions when they might appear without having to edit and maintain that repo file.  That is a reason that an rsync solution would be better.

I understand the point about the CDN, but would that explain that the RPM is available on the site but the index file is not updated?

Hi,

Edge servers are caching this directory listing for too long. We did contact the CDN to see if TTL can be reduced for those directory listings but is really not much we can do about it other than that.

Thanks
Bogdan

Bug 96174 is marked as duplicate of this bug

p.s. TTL on these are now set to 15min

Although this bug is marked as "Not a Bug", I think I should still notice people reading this bug report that, setting TTL to 15min seems not working for resolving this issue: some CDN nodes are still giving seriously out-of-date index pages. For example:

> curl -v https://repo.mysql.com/yum/mysql-connectors-community/el/ --resolve repo.mysql.com:443:23.36.253.13
* Added repo.mysql.com:443:23.36.253.13 to DNS cache
* Hostname repo.mysql.com was found in DNS cache
*   Trying 23.36.253.13:443...
* Connected to repo.mysql.com (23.36.253.13) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*  CAfile: /etc/ssl/certs/ca-certificates.crt
*  CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES256-GCM-SHA384
* ALPN, server accepted to use http/1.1
* Server certificate:
*  subject: C=US; ST=California; L=Redwood City; O=Oracle Corporation; CN=cdn.mysql.com
*  start date: Sep 15 00:00:00 2022 GMT
*  expire date: Sep 15 23:59:59 2023 GMT
*  subjectAltName: host "repo.mysql.com" matched cert's "repo.mysql.com"
*  issuer: C=US; O=DigiCert Inc; OU=www.digicert.com; CN=GeoTrust RSA CA 2018
*  SSL certificate verify ok.
> GET /yum/mysql-connectors-community/el/ HTTP/1.1
> Host: repo.mysql.com
> User-Agent: curl/7.74.0
> Accept: */*
> 
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< Accept-Ranges: bytes
< Content-Type: text/html
< Server: AkamaiNetStorage
< Content-Length: 903
< Date: Tue, 17 Jan 2023 17:46:00 GMT
< Connection: keep-alive
< 
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2 Final//EN">
<HTML>
 <HEAD>
  <TITLE>Index of /232905/yum/mysql-connectors-community/el</TITLE>
 </HEAD>
 <BODY>
<H1>Index of /232905/yum/mysql-connectors-community/el</H1>
<PRE>   Name                              Last modified        Size  
<HR>
<IMG SRC="/icons/dir.gif" ALT="[DIR]"> <A HREF="../">Parent Directory</A>                  01-Jan-1970 00:00      -  
<IMG SRC="/icons/dir.gif" ALT="[DIR]"> <A HREF="5/">5/</A>                                28-Apr-2017 17:19      -  
<IMG SRC="/icons/dir.gif" ALT="[DIR]"> <A HREF="6/">6/</A>                                25-Apr-2022 12:13      -  
<IMG SRC="/icons/dir.gif" ALT="[DIR]"> <A HREF="7/">7/</A>                                25-Apr-2022 12:13      -  
<IMG SRC="/icons/dir.gif" ALT="[DIR]"> <A HREF="8/">8/</A>                                25-Apr-2022 12:13      -  
</PRE><HR>
</BODY></HTML>
* Connection #0 to host repo.mysql.com left intact

The directory "9" is not in this index page. Curiously, accessing this page with HTTP gives correct index page:

> curl -v http://repo.mysql.com/yum/mysql-connectors-community/el/ --resolve repo.mysql.com:80:23.36.253.13
* Added repo.mysql.com:80:23.36.253.13 to DNS cache
* Hostname repo.mysql.com was found in DNS cache
*   Trying 23.36.253.13:80...
* Connected to repo.mysql.com (23.36.253.13) port 80 (#0)
> GET /yum/mysql-connectors-community/el/ HTTP/1.1
> Host: repo.mysql.com
> User-Agent: curl/7.74.0
> Accept: */*
> 
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< Accept-Ranges: bytes
< Content-Type: text/html
< Server: AkamaiNetStorage
< Content-Length: 1020
< Date: Tue, 17 Jan 2023 17:49:53 GMT
< Connection: keep-alive
< 
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2 Final//EN">
<HTML>
 <HEAD>
  <TITLE>Index of /232905/yum/mysql-connectors-community/el</TITLE>
 </HEAD>
 <BODY>
<H1>Index of /232905/yum/mysql-connectors-community/el</H1>
<PRE>   Name                              Last modified        Size  
<HR>
<IMG SRC="/icons/dir.gif" ALT="[DIR]"> <A HREF="../">Parent Directory</A>                  01-Jan-1970 00:00      -  
<IMG SRC="/icons/dir.gif" ALT="[DIR]"> <A HREF="5/">5/</A>                                28-Apr-2017 17:19      -  
<IMG SRC="/icons/dir.gif" ALT="[DIR]"> <A HREF="6/">6/</A>                                14-Jan-2023 08:45      -  
<IMG SRC="/icons/dir.gif" ALT="[DIR]"> <A HREF="7/">7/</A>                                14-Jan-2023 08:46      -  
<IMG SRC="/icons/dir.gif" ALT="[DIR]"> <A HREF="8/">8/</A>                                14-Jan-2023 08:46      -  
<IMG SRC="/icons/dir.gif" ALT="[DIR]"> <A HREF="9/">9/</A>                                14-Jan-2023 08:46      -  
</PRE><HR>
</BODY></HTML>
* Connection #0 to host repo.mysql.com left intact

I guess that port 80 and 443 connect to different CDN nodes even they share the same IP address. 

And it seems that setting proper HTTP header for directory listing may help CDN do caching correctly?