MySQL Bugs: #111159: Incomplete documentation on MySQL rpm rebuilds makes rebuilding packages hard

Bug #111159	Incomplete documentation on MySQL rpm rebuilds makes rebuilding packages hard
Submitted:	26 May 2023 6:02	Modified:	4 Nov 2023 9:47
Reporter:	Simon Mudd (OCA)	Email Updates:
Status:	Verified	Impact on me:	None
Category:	MySQL Server: Compiling	Severity:	S3 (Non-critical)
Version:	8.0	OS:	CentOS
Assigned to:		CPU Architecture:	Any
Tags:	build, reproduce, rpm

Description:
These days reproducible builds are very much in fashion. This is good as it means you can always define the environment and then build something as desired. This also means others can reproduce the work you are doing.

I use rpm packages for running MySQL and have sometimes built custom plugins for various use cases. Often the plugins are quite simple.

I want to be able to reproduce the build process used by Oracle to enough of an extent that I can build the binary rpms from the src rpm packages provided. The intention of src rpm packages is that they provide the source software and you can reproducibly build the binaries from this.

I have been trying to do this with CentOS 8 builds and I have come up with a process to do this. It works for 8.0.30..8.0.33. 8.0.26 fails as using the same process generates an error.

I am experiencing similar issues when trying to rebuild binary rpms for CentOS 7 and CentOS 9.

Basically the MySQL documentation / Build requires configuration of the rpm packages does not provide full information on achieving a reproducible build.

I decided to try to build a reproducible build system for this process.
The results are here: https://github.com/sjmudd/mysql-rpm-builder

Hopefully the documentation and usage is straight forward.
You'll see there are some "odd hacks" required to get some of the build process to work. It worked for me for those versions I was able to build, but perhaps there's a better way to do this.

How to repeat:
I decided to try to figure this out myself. See: https://github.com/sjmudd/mysql-rpm-builder

If you run the build system and have docker this works as indicated if you do something like:

docker run --rm -it --network=host --hostname=mysql-builder -v $PWD:/data quay.io/centos/centos:stream8 /data/build 8.0.33

but fails with:

docker run --rm -it --network=host --hostname=mysql-builder -v $PWD:/data quay.io/centos/centos:stream8 /data/build 8.0.26

Clearly the versions were built at different times and the "OS prepare phase" tries to address these differences but I have not obviously figured out correctly how to do this.

The end result is I am unable to build all the MySQL versions I would like
and given this tooling will also allow me to patch the versions provided by Oracle I can not patch the rpms to produce the required plugins in the way I want.

The alternative approach would be to build from the github src tree but it's not 100% clear to me the specific process to do that and it may be that this also has the issue of not explicitly specifying the full build environment for the build to work.

Suggested fix:
Ideally:
- modify documentation to explain the required build process
- modify the mysql.spec files to be more explicit in required packages needed to build the binary rpms

If you want provide PR with improvements to my own repo so that the information is available there publicly.

To be clear. This should be really easy:

1. download src.rpm from Oracle for the specific MySQL version
2. install src rpm with rpm -ivh mysql-community-8.0.XX-1.elX.src.rpm
2. install build dependencies
3. build the binary packages with rpmbuild --define 'rh8 1' -ba mysql.spec from the ~/rpmbuild/SPECS directory.

In practice:
- configuration of additional external repos is needed (e.g. powertools)
- configuration of tooling for the build tools / gcc is needed (e.g gcc-toolset-XX)
- seems some patching of the toolset is needed for builds to complete as otherwise there are missing libraries
- ideally you build from a clean, minimal system and define exactly what's in that build environment.

Hi Mr. Mudd,

Thank you for your bug report.

There seems to be missing some documentation.

We think that this is mostly a documentation bug, but we shall keep the category as you reported it.

Verified as reported.

ok, yes, indeed. The compilation does not fail. The final stage of the rpmbuild process fails.

I'll upload the logging from an earlier (failed) run where I saw this. I've modified the formatting but you should get the idea.

logging of installing required rpm to handle the build

Attachment: prepare__8.0.26__centos.8__20230525.220656.log.gz (application/x-gzip, text), 20.20 KiB.

build process, interesting bit at then end where there are missing files

Attachment: build__8.0.26__centos.8__20230525.221331.log.gz (application/x-gzip, text), 778.45 KiB.

Again the rpmbuild process fails, the code does compile. This is due to the spec file's file list not matching the files which are put in the package.

I do agree this is more a documentation bug as with the right configuration, and I'm clearly missing something the build process should work.

That's either ensuring more "required, but not documented as required" packages are present on the build system or ensuring that some --define "name <value>" setting is added to trigger the build process to behave slightly differently.

However, since I can see quite clearly that a lot of this information varies between MySQL releases even on the same OS it would be good if the documentation or the git repo itself makes these requirements explicit as I'd like to know that I can build on in theory any rpm based OS supported by Oracle and know how to do that, even if my current interest is CentOS 7..9

Thank you, Mr. Mudd,

All your comments are copied to our internal bugs database.

There is an other generic problem here regarding rebuilding.

8.0.26 is a long time ago, to make perfect rebuild you need all tools
to be at the version of the time 8.0.26 was built.

For example you need GCC 10 package to be exact version and release from
May/June 2021. The GCC 10 you get by 'yum install ...' today is something different.

Other issue is that 8.0.26 was never built on EL9. 
Initial release for 8.0.30 foe EL9. 

Also note that use of CentOS Stream is not supported.

Hi Terje Røsten,

- Your points are completely valid, and clearly my intention was to move forward.
- If your builds work against OEL or RHEL version X that's absolutely fine. I can figure out differences to take that into account.
- it's also clear that when building packages it may be historically convenient to record what the build environment was. In general with RPMS that's never been done leading to many issues this specific bug report is referring to.

This came up with the desire to "be able to" rebuild an existing rpm exclusively from the src.rpms you provide. If I can do that in a repeatable manner I can then make changes.

You might argue that I should build from source, from the github git tree but there are no explicit instructions on how to setup the environment to do that.

Clearly my focus here is on rpms. You support a large number of platforms and architectures and so the work you have to do is much more complex than the tighter focus I'm interested in.

8.0.33 has come out, 8.0.34 is out soon as will be 8.1. So I'd like to be able to rebuild and perhaps copy some patches from one version to another. With 8.0 that's been straight forward as for example the plugin interface has barely changed. I expect this will mean that I won't be able to use existing 8.0 plugins on 8.1 so I will have to rebuild them, even if the interface is in theory unchanged. I believe the major version is part of that.

So again what would be convenient would be to have enough documentation to rebuild from a src.rpm the "current version" of MySQL 8.0 on CentOS 8/9.

I sort of started in the middle and tried to work forwards and backwards but bumped into different issues as the tool chains change and I even had to do some weird hacking of symlinks to fix some weird issues. As far as I could see some similar adjustments might be necessary in your own environment but that's clearly opaque to someone from outside.

So if you can improve documentation here for 8.0 / 8.1 latest versions that would be really useful. Building from a docker image is these days "the easiest way to do things" and if all you provide is something built from OEL 8 / 9 that would be great.

Update on this now that 8.1.0 and 8.0.34 have been released.
- I notice that https://dev.mysql.com/doc/refman/8.1/en/installing-source-distribution.html provides a very minimalistic definition of what's needed to do a build, nor does it mention the required build dependencies.

so: rpmbuild --rebuild --clean MySQL-VERSION.src.rpm is I think highly likely to fail for a number of reasons.

As mentioned the GCC compiler set which has been used on each Oracle-built RHEL release (and even amongst similar versions) may change. This is confusing, not documented, not part of the BuildRequires: setup.

8.0.34 uses gcc-toolset-12
8.0.33 uses gcc-toolset-12
8.0.32 uses gcc-toolset-11
...
8.0.26 uses gcc-toolset-10

None of this is documented.

Terje Røsten, you wrote: "Also note that use of CentOS Stream is not supported".

This is a community bug reporting system so I do not expect support. However, it would be useful to know what OS you built your "RHEL" binaries on and it would be convenient to know how you did this, so that someone using exactly the same OS can reproduce your process. Clearly other "compatible" OSes may work, but that's a different discussion though up to now the pre-built packages you have provided for RHEL have worked fine on CentOS.

Basically I think it would be useful to have more complete information BuildRequires: list, dependent on the specific MySQL and OS version used.

e.g. for CentOS 8 / RHEL 8 I think you're missing in your BuildRequires:

bind-utils
bison
cmake
cyrus-sasl-devel
libaio-devel
gcc-toolset-12-annobin-annocheck
gcc-toolset-12-annobin-plugin-gcc
gcc-toolset-12-binutils
gcc-toolset-12-gcc
gcc-toolset-12-gcc-c++
libcurl-devel
libtirpc-devel
libudev-devel
ncurses-devel
numactl-devel
openldap-devel
openssl-devel
perl
perl-JSON
rpcgen
time

Without installing these rpms first rpmbuild will fail.

If you're willing to accept a patch for MySQL 8.0/8.1 I can provide that and also for other OS versions, e.g. RHEL/OEL 9

I also bumped into similar issues trying to build from source on ubuntu 22.04 LTS. I guess that would need a different bug report, but the same underlying problem exists: a clear to follow process of building MySQL requires the dependencies to be specified and the process to follow to be described. I had some issues setting up the build environment correctly and the build failed as a consequence. So while most people do not build their own MySQL servers there are still a few that do and providing good docs on what's required to complete this successfully would be most helpful.

sorry I included the full list of packages required from the CentOS 8 stream docker image. What seems to be missing is this:

bind-utils
gcc-toolset-12-annobin-annocheck
gcc-toolset-12-annobin-plugin-gcc
gcc-toolset-12-binutils
gcc-toolset-12-gcc
gcc-toolset-12-gcc-c++
libudev-devel
perl-JSON

Apologies.

It looks like I must have been misreading the mysql.spec.in file as looking in more detail the required rpms are configured in the BuildRequires: section if you use --define 'el8 1' (or 'el9 1') as appropriate.

Hi Mr. Mudd,

Thank you for your feedback.

After having tried to compile the src rpm for OL9, it seems the spec file needs to have some modifications related to BuildRequires sections.

The two lines below should be moved outside the %if 0%{?commercial} section as they are required for clients plugins too (like authentication_kerberos_client.so):

BuildRequires:  krb5-devel
BuildRequires:  libcurl-devel

And to be able to build all the packages (debug included) those packages are also required:

libcap-devel
libcbor-devel
ibudev-devel
libfido2-devel

With these dependencies, doing rpmbuild --rebuild mysql-community-8.0.34-1.el9.src.rpm will work.

Cheers.

(*)libudev-devel

Thank you, Fred.

After having discussed with Simon, it seems that the following two packages are not required to BUILD:
* libcap-devel
* libcbor-devel

Will test and add a comment.

I confirm that these two devel packages are not required to build the sources

Thank you, Fred.

note: See: https://github.com/sjmudd/mysql-rpm-builder has been updated and now works successfully against CentOS 8/9, OEL 8/9 for latest 8.0 / 8.1 / 8.2 versions.

Thank you for your contribution.