Bug #113045 Build failure with Protobuf 24.2 in Linux
Submitted: 10 Nov 2023 14:54 Modified: 15 Nov 2023 2:41
Reporter: gordon wang Email Updates:
Status: Verified Impact on me:
Category:MySQL Server: Compiling Severity:S3 (Non-critical)
Version:8.0 OS:Linux
Assigned to: CPU Architecture:Any (x86_64,arm64)

[10 Nov 2023 14:54] gordon wang

In Linux, if you compile mysql-8.0.35 with protobuf 24.2, the error "/usr/bin/ld: /usr/lib64/libprotoc.so: undefined reference to symbol  '_ZN4absl12lts_2023080213base_internal12SpinLockWaitEPSt6atomicIjEiPKNS1_22SpinLockWaitTransitionENS1_14SchedulingModeE'

/usr/bin/ld: /usr/lib64/libabsl_spinlock_wait.so.2308.0.0: error adding symbols: DSO missing from command line.

According to the release notes of mysql-8.0.35, this issue should have been fixed in the current release, but this is not the case.

“Building with WITH_PROTOBUF=system failed with Protobuf 22 or newer due to Protobuf 22 adding the Abseil dependency.  (Bug #111469, Bug #111623, Bug #35546459, Bug #35550389)”

Reading the mysql source code, it seems that protobuf 22+ is only fixed on MacOS, so the problem still exists on Linux.

How to repeat:
How to reproduce:

Environment: Linux x86-64/arm64 platform

Dependency: protobuf 24.2

Compile: Open -DWITH_PROTOBUF="system", then compile the mysql-8.0.35 code

Suggested fix:
My fix:

1. In cmake/protobuf.cmake, add fixes related to IF(LINUX AND WITH_PROTOBUF STREQUAL "system" AND PB_MINOR_VERSION VERSION_GREATER 21)

2. In cmake/fileutils.cmake, improve the FIND_OBJECT_DEPENDENCIES function in Linux.

The git link to my fix code is as follows:
[10 Nov 2023 15:55] gordon wang
Question 2:

Following the previous question, I found a new one, described as follows:

Run the FIND_OBJECT_DEPENDENCIES function to find dependent libraries. If the output of objdump is as follows:

NEEDED               libstdc++.so.6
NEEDED               libc.so.6
NEEDED ld-Linux-x86-64.2 ".

At this point, libc.so.6 can be processed normally and converted to "-lc";
However, "libstdc++.so.6" cannot be captured by the regular expression because it contains "++", so it cannot install the transposition "-lstdc++";
Although "ld-Linux-x86-64.2" can be captured by the regular expression, it is incorrectly converted to "-ld-Linux-x86-64", and the correct result should be "-ldl".

Function output error results are as follows:

-labsl_spinlock_wait -llibstdc -lgcc_s -lc -lld-linux-x86-64

My fix code is here: https://github.com/mysql/mysql-server/pull/504
[13 Nov 2023 10:23] MySQL Verification Team
Hi Mr. wang,

Thank you for your bug report.

However, for the building of MySQL you have to use protobuf version that is BUNDLED with our source code.

Hence, if you do not follow that, you will hit the problems like the ones that you describe.

Let us know how did it go with the bundled protobuf.
[13 Nov 2023 10:25] MySQL Verification Team

One more comment.

This is the only version and release of the protobuf that we support in our builds.

Hence, we can not accept patches for some other release.
[14 Nov 2023 3:06] gordon wang
Your answer involves two questions, please let me reply one by one.

1. Are there any problems with bundled protobuf?
bundled protobuf is version 19.x, this version of absl can be linked normally, so there is no problem.

2. You can not accept patches for some other release.
To correct my description, this compilation problem exists in all versions of protobuf 22+ (not just protobuf 24.2).

As far as I know, a user in #111469 reported a compilation problem with protobuf 22+ on MacOS to mysql. Bug link below: https://bugs.mysql.com/bug.php?id=111469

Then mysql fixed protobuf compilation issues for MacOS in 8.0.35, 8.2.0. Does this mean that mysql needs to consider protobuf 22+ fixes?

Then I discovered that protobuf 22+ compilation issues also exist on Linux (yes, you only fixed MacOS).
Now that you fixed the issue on MacOS, do you need to fix it on Linux?

Looking forward to your reply~
[14 Nov 2023 10:14] MySQL Verification Team
Hi Mr. wang,

Thank you for your discussion.

We shall definitely consider fixing this build problem on Linux as well.

This bug is now verified.
[14 Nov 2023 13:42] gordon wang
We now have an agreement on whether to fix protobuf 22+ on Linux.
Next, we can discuss what can be fixed. It would be even better if I could talk to the mysql developers about the fix.

In fact, mysql has two areas where the code needs to be changed:
1. In cmake/protobuf.cmake, support protobuf 22+ on Linux;
2. In cmake/fileutils.cmake, the FIND_OBJECT_DEPENDENCIES function in Linux needs to be improved (this function does not handle some so libraries printed by objdump correctly)

In my PR, the recurrence and repair of these problems are explained in more detail. I recommend that mysql developers look directly at my PR for a clearer description.
My PR links is as follows: https://github.com/mysql/mysql-server/pull/504

By the way, does the mysql community not accept code from regular developers like me?
I am willing to adjust my code based on the input of the mysql community if needed.
If my PR is accepted, I will be very happy.
[14 Nov 2023 14:47] MySQL Verification Team
HI Mr. Wang,

First of all, thanks again for your contribution.

We are very happy to accept patches from anyone in the community, who wishes to contribute. You can not discuss your case with developers, but our Team is quite capable of analysing your contribution and your idea.  That does not mean that the final patch will be exactly the same as the one that you proposed.

We think that your suggestions are on the right track and we shall forward them to our Development Team.
[15 Nov 2023 2:41] gordon wang
Thank you for your team's recognition.

I can adjust my patch code if necessary.
Looking forward to contributing to the mysql code.