Bug #111469 Build failure with Protobuf 22+
Submitted: 17 Jun 2023 8:07 Modified: 14 Aug 2023 22:15
Reporter: Carlo Cabrera Email Updates:
Status: Closed Impact on me:
None 
Category:MySQL Server: Compiling Severity:S2 (Serious)
Version:8.0.33 OS:Any
Assigned to: CPU Architecture:Any

[17 Jun 2023 8:07] Carlo Cabrera 
Description:
Building MySQL 8.0.33 with `-DWITH_PROTOBUF=system` will fail to build when the system Protobuf is version 22 or newer.

There appear to be at least two changes to handle:
1. The build needs to pass additional linker flags to link to Abseil, which Protobuf now pulls in as a dependency. This can be done via `pkg-config` or using Protobuf's `protobuf-config.cmake`. Importantly, using CMake's builtin `FindProtobuf` module will *not* work.
2. Code that uses `google::protobuf::LogHandler` will need to be adapted to use Abseil's logging code, since Protobuf has done the same.

There may be other changes required.

See:
https://github.com/protocolbuffers/protobuf/commit/a9f1ea6371c108876649f27a5940a59cc859476...
https://github.com/protocolbuffers/protobuf/issues/12292

You can also find build logs from various attempts to build MySQL against the latest version of Protobuf at:
https://github.com/Homebrew/homebrew-core/actions/runs/5296742952/jobs/9588107254?pr=13400...
https://github.com/Homebrew/homebrew-core/actions/runs/5296742952/jobs/9588107254?pr=13400...

How to repeat:
Build MySQL Server against Protobuf 22 or newer.
[28 Jun 2023 10:10] MySQL Verification Team 
Hello Carlo Cabrera,

Thank you for the report and feedback.
Verified as described.

regards,
Umesh
[30 Jun 2023 12:38] Tor Didriksen 
Posted by developer:
 
for reference, the abseil libraries to link:

$otool -L /usr/local/lib/libprotobuf-lite.dylib 
/usr/local/lib/libprotobuf-lite.dylib:
        /usr/local/opt/protobuf/lib/libprotobuf-lite.23.3.0.dylib (compatibility version 0.0.0, current version 23.3.0)
        /usr/local/opt/abseil/lib/libabsl_die_if_null.2301.0.0.dylib (compatibility version 2301.0.0, current version 0.0.0)
        /usr/local/opt/abseil/lib/libabsl_flags.2301.0.0.dylib (compatibility version 2301.0.0, current version 0.0.0)
        /usr/local/opt/abseil/lib/libabsl_log_initialize.2301.0.0.dylib (compatibility version 2301.0.0, current version 0.0.0)
        /usr/local/opt/abseil/lib/libabsl_statusor.2301.0.0.dylib (compatibility version 2301.0.0, current version 0.0.0)
        /usr/local/opt/abseil/lib/libabsl_log_internal_check_op.2301.0.0.dylib (compatibility version 2301.0.0, current version 0.0.0)
        /usr/local/opt/abseil/lib/libabsl_leak_check.2301.0.0.dylib (compatibility version 2301.0.0, current version 0.0.0)
        /usr/local/opt/abseil/lib/libabsl_log_internal_conditions.2301.0.0.dylib (compatibility version 2301.0.0, current version 0.0.0)
        /usr/local/opt/abseil/lib/libabsl_log_internal_message.2301.0.0.dylib (compatibility version 2301.0.0, current version 0.0.0)
        /usr/local/opt/abseil/lib/libabsl_log_internal_nullguard.2301.0.0.dylib (compatibility version 2301.0.0, current version 0.0.0)
        /usr/local/opt/abseil/lib/libabsl_examine_stack.2301.0.0.dylib (compatibility version 2301.0.0, current version 0.0.0)
        /usr/local/opt/abseil/lib/libabsl_log_internal_format.2301.0.0.dylib (compatibility version 2301.0.0, current version 0.0.0)
        /usr/local/opt/abseil/lib/libabsl_log_internal_proto.2301.0.0.dylib (compatibility version 2301.0.0, current version 0.0.0)
        /usr/local/opt/abseil/lib/libabsl_log_internal_log_sink_set.2301.0.0.dylib (compatibility version 2301.0.0, current version 0.0.0)
        /usr/local/opt/abseil/lib/libabsl_log_sink.2301.0.0.dylib (compatibility version 2301.0.0, current version 0.0.0)
        /usr/local/opt/abseil/lib/libabsl_log_entry.2301.0.0.dylib (compatibility version 2301.0.0, current version 0.0.0)
        /usr/local/opt/abseil/lib/libabsl_flags_internal.2301.0.0.dylib (compatibility version 2301.0.0, current version 0.0.0)
        /usr/local/opt/abseil/lib/libabsl_flags_marshalling.2301.0.0.dylib (compatibility version 2301.0.0, current version 0.0.0)
        /usr/local/opt/abseil/lib/libabsl_flags_reflection.2301.0.0.dylib (compatibility version 2301.0.0, current version 0.0.0)
        /usr/local/opt/abseil/lib/libabsl_flags_config.2301.0.0.dylib (compatibility version 2301.0.0, current version 0.0.0)
        /usr/local/opt/abseil/lib/libabsl_flags_program_name.2301.0.0.dylib (compatibility version 2301.0.0, current version 0.0.0)
        /usr/local/opt/abseil/lib/libabsl_flags_private_handle_accessor.2301.0.0.dylib (compatibility version 2301.0.0, current version 0.0.0)
        /usr/local/opt/abseil/lib/libabsl_flags_commandlineflag.2301.0.0.dylib (compatibility version 2301.0.0, current version 0.0.0)
        /usr/local/opt/abseil/lib/libabsl_flags_commandlineflag_internal.2301.0.0.dylib (compatibility version 2301.0.0, current version 0.0.0)
        /usr/local/opt/abseil/lib/libabsl_log_globals.2301.0.0.dylib (compatibility version 2301.0.0, current version 0.0.0)
        /usr/local/opt/abseil/lib/libabsl_log_internal_globals.2301.0.0.dylib (compatibility version 2301.0.0, current version 0.0.0)
        /usr/local/opt/abseil/lib/libabsl_hash.2301.0.0.dylib (compatibility version 2301.0.0, current version 0.0.0)
        /usr/local/opt/abseil/lib/libabsl_city.2301.0.0.dylib (compatibility version 2301.0.0, current version 0.0.0)
        /usr/local/opt/abseil/lib/libabsl_low_level_hash.2301.0.0.dylib (compatibility version 2301.0.0, current version 0.0.0)
        /usr/local/opt/abseil/lib/libabsl_raw_hash_set.2301.0.0.dylib (compatibility version 2301.0.0, current version 0.0.0)
        /usr/local/opt/abseil/lib/libabsl_hashtablez_sampler.2301.0.0.dylib (compatibility version 2301.0.0, current version 0.0.0)
        /usr/local/opt/abseil/lib/libabsl_status.2301.0.0.dylib (compatibility version 2301.0.0, current version 0.0.0)
        /usr/local/opt/abseil/lib/libabsl_cord.2301.0.0.dylib (compatibility version 2301.0.0, current version 0.0.0)
        /usr/local/opt/abseil/lib/libabsl_cordz_info.2301.0.0.dylib (compatibility version 2301.0.0, current version 0.0.0)
        /usr/local/opt/abseil/lib/libabsl_cord_internal.2301.0.0.dylib (compatibility version 2301.0.0, current version 0.0.0)
        /usr/local/opt/abseil/lib/libabsl_cordz_functions.2301.0.0.dylib (compatibility version 2301.0.0, current version 0.0.0)
        /usr/local/opt/abseil/lib/libabsl_exponential_biased.2301.0.0.dylib (compatibility version 2301.0.0, current version 0.0.0)
        /usr/local/opt/abseil/lib/libabsl_cordz_handle.2301.0.0.dylib (compatibility version 2301.0.0, current version 0.0.0)
        /usr/local/opt/abseil/lib/libabsl_synchronization.2301.0.0.dylib (compatibility version 2301.0.0, current version 0.0.0)
        /usr/local/opt/abseil/lib/libabsl_graphcycles_internal.2301.0.0.dylib (compatibility version 2301.0.0, current version 0.0.0)
        /usr/local/opt/abseil/lib/libabsl_time.2301.0.0.dylib (compatibility version 2301.0.0, current version 0.0.0)
        /usr/local/opt/abseil/lib/libabsl_civil_time.2301.0.0.dylib (compatibility version 2301.0.0, current version 0.0.0)
        /usr/local/opt/abseil/lib/libabsl_time_zone.2301.0.0.dylib (compatibility version 2301.0.0, current version 0.0.0)
        /System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation (compatibility version 150.0.0, current version 1971.0.0)
        /usr/local/opt/abseil/lib/libabsl_crc_cord_state.2301.0.0.dylib (compatibility version 2301.0.0, current version 0.0.0)
        /usr/local/opt/abseil/lib/libabsl_crc32c.2301.0.0.dylib (compatibility version 2301.0.0, current version 0.0.0)
        /usr/local/opt/abseil/lib/libabsl_crc_internal.2301.0.0.dylib (compatibility version 2301.0.0, current version 0.0.0)
        /usr/local/opt/abseil/lib/libabsl_crc_cpu_detect.2301.0.0.dylib (compatibility version 2301.0.0, current version 0.0.0)
        /usr/local/opt/abseil/lib/libabsl_bad_optional_access.2301.0.0.dylib (compatibility version 2301.0.0, current version 0.0.0)
        /usr/local/opt/abseil/lib/libabsl_stacktrace.2301.0.0.dylib (compatibility version 2301.0.0, current version 0.0.0)
        /usr/local/opt/abseil/lib/libabsl_str_format_internal.2301.0.0.dylib (compatibility version 2301.0.0, current version 0.0.0)
        /usr/local/opt/abseil/lib/libabsl_strerror.2301.0.0.dylib (compatibility version 2301.0.0, current version 0.0.0)
        /usr/local/opt/abseil/lib/libabsl_symbolize.2301.0.0.dylib (compatibility version 2301.0.0, current version 0.0.0)
        /usr/local/opt/abseil/lib/libabsl_malloc_internal.2301.0.0.dylib (compatibility version 2301.0.0, current version 0.0.0)
        /usr/local/opt/abseil/lib/libabsl_debugging_internal.2301.0.0.dylib (compatibility version 2301.0.0, current version 0.0.0)
        /usr/local/opt/abseil/lib/libabsl_demangle_internal.2301.0.0.dylib (compatibility version 2301.0.0, current version 0.0.0)
        /usr/local/opt/abseil/lib/libabsl_bad_variant_access.2301.0.0.dylib (compatibility version 2301.0.0, current version 0.0.0)
        /usr/local/opt/abseil/lib/libabsl_strings.2301.0.0.dylib (compatibility version 2301.0.0, current version 0.0.0)
        /usr/local/opt/abseil/lib/libabsl_throw_delegate.2301.0.0.dylib (compatibility version 2301.0.0, current version 0.0.0)
        /usr/local/opt/abseil/lib/libabsl_int128.2301.0.0.dylib (compatibility version 2301.0.0, current version 0.0.0)
        /usr/local/opt/abseil/lib/libabsl_strings_internal.2301.0.0.dylib (compatibility version 2301.0.0, current version 0.0.0)
        /usr/local/opt/abseil/lib/libabsl_base.2301.0.0.dylib (compatibility version 2301.0.0, current version 0.0.0)
        /usr/local/opt/abseil/lib/libabsl_raw_logging_internal.2301.0.0.dylib (compatibility version 2301.0.0, current version 0.0.0)
        /usr/local/opt/abseil/lib/libabsl_log_severity.2301.0.0.dylib (compatibility version 2301.0.0, current version 0.0.0)
        /usr/local/opt/abseil/lib/libabsl_spinlock_wait.2301.0.0.dylib (compatibility version 2301.0.0, current version 0.0.0)
        /usr/lib/libc++.1.dylib (compatibility version 1.0.0, current version 1500.65.0)
        /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1319.100.3)
[2 Jul 2023 14:35] Carlo Cabrera 
Note that Apple ld64 is often a little overzealous when it comes to linking libraries because of the lack of support for `--as-needed`.

Linking the following Abseil libraries may suffice:
```
❯ objdump -p libprotobuf-lite.so | rg NEEDED
  NEEDED       libabsl_log_internal_check_op.so.2301.0.0
  NEEDED       libabsl_log_internal_message.so.2301.0.0
  NEEDED       libabsl_log_internal_nullguard.so.2301.0.0
  NEEDED       libabsl_hash.so.2301.0.0
  NEEDED       libabsl_raw_hash_set.so.2301.0.0
  NEEDED       libabsl_cord.so.2301.0.0
  NEEDED       libabsl_synchronization.so.2301.0.0
  NEEDED       libabsl_strings.so.2301.0.0
  NEEDED       libabsl_throw_delegate.so.2301.0.0
  NEEDED       libstdc++.so.6
  NEEDED       libgcc_s.so.1
  NEEDED       libc.so.6
  NEEDED       ld-linux-x86-64.so.2
```
[3 Jul 2023 8:25] Tor Didriksen 
Posted by developer:
 
objdump -p on mac has different output than on Linux, at least for me

.....
Load command 11
      cmd LC_SOURCE_VERSION
  cmdsize 16
  version 0.0
Load command 12
          cmd LC_LOAD_DYLIB
      cmdsize 88
         name /usr/local/opt/abseil/lib/libabsl_die_if_null.2301.0.0.dylib (offset 24)
   time stamp 2 Thu Jan  1 01:00:02 1970
      current version 0.0.0
compatibility version 2301.0.0
Load command 13
          cmd LC_LOAD_DYLIB
      cmdsize 80
         name /usr/local/opt/abseil/lib/libabsl_flags.2301.0.0.dylib (offset 24)
   time stamp 2 Thu Jan  1 01:00:02 1970
      current version 0.0.0
compatibility version 2301.0.0
....
....
[7 Jul 2023 9:53] Carlo Cabrera 
Yes, that should be expected since the structure of the ELF and MachO headers differ.

My suggestion was merely that the libraries that are linked on Linux are probably a more accurate representation of the libraries that are needed because new versions of GNU ld are less eager than Apple ld64 when it comes to linking libraries.
[7 Jul 2023 10:09] Tor Didriksen 
Posted by developer:
 
It turns out that the code using google::protobuf::LogHandler
can simply be removed, it was used for internal debugging only.

I experimented with manual linking of needed abseil libs, something like

    FIND_PACKAGE(absl REQUIRED)
    INCLUDE("${absl_DIR}/abslTargets.cmake")
    UNSET(ABSL_LIBS)
    LIST(APPEND ABSL_LIBS absl::base)
    LIST(APPEND ABSL_LIBS absl::int128)
    LIST(APPEND ABSL_LIBS absl::log_internal_check_op)
    LIST(APPEND ABSL_LIBS absl::log_internal_message)
    LIST(APPEND ABSL_LIBS absl::log_severity)
    LIST(APPEND ABSL_LIBS absl::raw_logging_internal)
    LIST(APPEND ABSL_LIBS absl::spinlock_wait)
    LIST(APPEND ABSL_LIBS absl::strings)
    LIST(APPEND ABSL_LIBS absl::strings_internal)
    LIST(APPEND ABSL_LIBS absl::throw_delegate)

But I ended up using 'otool -L' and filtering the output.
That should be more future proof, and simpler to maintain.
[14 Aug 2023 22:15] Philip Olson 
Posted by developer:
 
Fixed as of the upcoming MySQL Server 8.0.35 and 8.2.0 releases, and here's the proposed changelog entry from the documentation team:

On macOS, building with WITH_PROTOBUF=system fails with Protobuf 22 or
newer, which is installed with Homebrew, due to Protobuf 22 adding the
Abseil dependency.

Thank you for the bug report.