MySQL Bugs: #101231: Min required Protobuf version not properly detected

Bug #101231	Min required Protobuf version not properly detected
Submitted:	19 Oct 2020 16:56	Modified:	10 Nov 2020 8:37
Reporter:	Georgi Sotirov	Email Updates:
Status:	Closed	Impact on me:	None
Category:	MySQL Server: Compiling	Severity:	S3 (Non-critical)
Version:	8.0	OS:	Linux (Slackware)
Assigned to:		CPU Architecture:	x86

Description:
When trying to build MySQL Server 8.0.22 from source with GCC 9.3.0, CMake 3.18.4 and Protobuf 3.2.1 on Slackware64-current with option -DWITH_PROTOBUF=system, I receive the following compilation errors:

[ 26%] Linking CXX executable ../../../../runtime_output_directory/xprotocol_plugin
cd /usr/src/tmp/mysql-8.0.22/build/plugin/x/protocol/plugin && /usr/bin/cmake -E cmake_link_script CMakeFiles/xprotocol_plugin.dir/link.txt --verbose=1
/usr/bin/g++ -std=c++14 -fno-omit-frame-pointer -ffp-contract=off -ftls-model=initial-exec -O3 -march=i586 -mtune=i686 -fPIC -Wall -Wextra -Wformat-security -Wvla -Wundef -Woverloaded-virtual -Wcast-qual -Wimplicit-fallthrough=2 -Wstringop-truncation -Wsuggest-override -Wlogical-op -DDBUG_OFF -ffunction-sections -fdata-sections -O3 -DNDEBUG  -fuse-ld=lld  -Wl,--no-as-needed -lpthread CMakeFiles/xprotocol_plugin.dir/message_field_chain.cc.o CMakeFiles/xprotocol_plugin.dir/messages_used_by_server.cc.o CMakeFiles/xprotocol_plugin.dir/xprotocol_plugin.cc.o -o ../../../../runtime_output_directory/xprotocol_plugin  -lpthread ../protobuf/libmysqlxmessages.a /usr/lib/libprotoc.so /usr/lib/libprotobuf.so -lpthread
ld.lld: error: undefined symbol: google::protobuf::compiler::CodeGenerator::GenerateAll(std::vector<google::protobuf::FileDescriptor const*, std::allocator<google::protobuf::FileDescriptor const*> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, google::protobuf::compiler::GeneratorContext*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >*) const
>>> referenced by xprotocol_plugin.cc
>>>               CMakeFiles/xprotocol_plugin.dir/xprotocol_plugin.cc.o:(vtable for XProtocol_plugin)

ld.lld: error: undefined symbol: google::protobuf::internal::WireFormatLite::WriteStringMaybeAliased(int, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, google::protobuf::io::CodedOutputStream*)
>>> referenced by mysqlx.pb.cc
>>>               mysqlx.pb.cc.o:(Mysqlx::Ok::SerializeWithCachedSizes(google::protobuf::io::CodedOutputStream*) const) in archive ../protobuf/libmysqlxmessages.a
>>> referenced by mysqlx.pb.cc
>>>               mysqlx.pb.cc.o:(Mysqlx::Error::SerializeWithCachedSizes(google::protobuf::io::CodedOutputStream*) const) in archive ../protobuf/libmysqlxmessages.a
>>> referenced by mysqlx.pb.cc
>>>               mysqlx.pb.cc.o:(Mysqlx::Error::SerializeWithCachedSizes(google::protobuf::io::CodedOutputStream*) const) in archive ../protobuf/libmysqlxmessages.a

ld.lld: error: undefined symbol: google::protobuf::io::CodedOutputStream::WriteStringWithSizeToArray(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unsigned char*)
>>> referenced by mysqlx.pb.cc
>>>               mysqlx.pb.cc.o:(Mysqlx::Ok::InternalSerializeWithCachedSizesToArray(bool, unsigned char*) const) in archive ../protobuf/libmysqlxmessages.a
>>> referenced by mysqlx.pb.cc
>>>               mysqlx.pb.cc.o:(Mysqlx::Error::InternalSerializeWithCachedSizesToArray(bool, unsigned char*) const) in archive ../protobuf/libmysqlxmessages.a
>>> referenced by mysqlx.pb.cc
>>>               mysqlx.pb.cc.o:(Mysqlx::Error::InternalSerializeWithCachedSizesToArray(bool, unsigned char*) const) in archive ../protobuf/libmysqlxmessages.a

ld.lld: error: undefined symbol: google::protobuf::internal::fixed_address_empty_string[abi:cxx11]
>>> referenced by mysqlx.pb.cc
>>>               mysqlx.pb.cc.o:(google::protobuf::internal::ArenaStringPtr::CreateInstanceNoArena(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const*) (.isra.0.constprop.0)) in archive ../protobuf/libmysqlxmessages.a
>>> referenced by mysqlx.pb.cc
>>>               mysqlx.pb.cc.o:(Mysqlx::Ok::SharedCtor()) in archive ../protobuf/libmysqlxmessages.a
>>> referenced by mysqlx.pb.cc
>>>               mysqlx.pb.cc.o:(Mysqlx::Ok::SharedDtor()) in archive ../protobuf/libmysqlxmessages.a
>>> referenced 9 more times

ld.lld: error: undefined symbol: google::protobuf::MessageFactory::InternalRegisterGeneratedFile(char const*, void (*)(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&))
>>> referenced by mysqlx.pb.cc
>>>               mysqlx.pb.cc.o:(Mysqlx::protobuf_mysqlx_2eproto::AddDescriptorsImpl()) in archive ../protobuf/libmysqlxmessages.a

ld.lld: error: undefined symbol: google::protobuf::internal::AssignDescriptors(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, google::protobuf::internal::MigrationSchema const*, google::protobuf::Message const* const*, unsigned int const*, google::protobuf::MessageFactory*, google::protobuf::Metadata*, google::protobuf::EnumDescriptor const**, google::protobuf::ServiceDescriptor const**)
>>> referenced by mysqlx.pb.cc
>>>               mysqlx.pb.cc.o:(Mysqlx::protobuf_mysqlx_2eproto::(anonymous namespace)::protobuf_AssignDescriptors()) in archive ../protobuf/libmysqlxmessages.a

ld.lld: error: undefined symbol: google::protobuf::internal::ArenaStringPtr::AssignWithDefault(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const*, google::protobuf::internal::ArenaStringPtr)
>>> referenced by mysqlx.pb.cc
>>>               mysqlx.pb.cc.o:(Mysqlx::Error::Error(Mysqlx::Error const&)) in archive ../protobuf/libmysqlxmessages.a
>>> referenced by mysqlx.pb.cc
>>>               mysqlx.pb.cc.o:(Mysqlx::Error::Error(Mysqlx::Error const&)) in archive ../protobuf/libmysqlxmessages.a
>>> referenced by mysqlx.pb.cc
>>>               mysqlx.pb.cc.o:(Mysqlx::Ok::MergeFrom(Mysqlx::Ok const&)) in archive ../protobuf/libmysqlxmessages.a
>>> referenced 3 more times

ld.lld: error: undefined symbol: google::protobuf::internal::WireFormatLite::ReadBytes(google::protobuf::io::CodedInputStream*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >*)
>>> referenced by mysqlx.pb.cc
>>>               mysqlx.pb.cc.o:(Mysqlx::Ok::MergePartialFromCodedStream(google::protobuf::io::CodedInputStream*)) in archive ../protobuf/libmysqlxmessages.a
>>> referenced by mysqlx.pb.cc
>>>               mysqlx.pb.cc.o:(Mysqlx::Error::MergePartialFromCodedStream(google::protobuf::io::CodedInputStream*)) in archive ../protobuf/libmysqlxmessages.a

ld.lld: error: undefined symbol: google::protobuf::Message::GetTypeName[abi:cxx11]() const
>>> referenced by mysqlx.pb.cc
>>>               mysqlx.pb.cc.o:(vtable for Mysqlx::ClientMessages) in archive ../protobuf/libmysqlxmessages.a
>>> referenced by mysqlx.pb.cc
>>>               mysqlx.pb.cc.o:(vtable for Mysqlx::ServerMessages) in archive ../protobuf/libmysqlxmessages.a
>>> referenced by mysqlx.pb.cc
>>>               mysqlx.pb.cc.o:(vtable for Mysqlx::Ok) in archive ../protobuf/libmysqlxmessages.a
>>> referenced 1 more times

ld.lld: error: undefined symbol: google::protobuf::Message::InitializationErrorString[abi:cxx11]() const
>>> referenced by mysqlx.pb.cc
>>>               mysqlx.pb.cc.o:(vtable for Mysqlx::ClientMessages) in archive ../protobuf/libmysqlxmessages.a
>>> referenced by mysqlx.pb.cc
>>>               mysqlx.pb.cc.o:(vtable for Mysqlx::ServerMessages) in archive ../protobuf/libmysqlxmessages.a
>>> referenced by mysqlx.pb.cc
>>>               mysqlx.pb.cc.o:(vtable for Mysqlx::Ok) in archive ../protobuf/libmysqlxmessages.a
>>> referenced 1 more times
collect2: error: ld returned 1 exit status
make[2]: *** [plugin/x/protocol/plugin/CMakeFiles/xprotocol_plugin.dir/build.make:136: runtime_output_directory/xprotocol_plugin] Error 1
make[2]: Leaving directory '/usr/src/tmp/mysql-8.0.22/build'
make[1]: *** [CMakeFiles/Makefile2:6228: plugin/x/protocol/plugin/CMakeFiles/xprotocol_plugin.dir/all] Error 2
make[1]: Leaving directory '/usr/src/tmp/mysql-8.0.22/build'
make: *** [Makefile:182: all] Error 2

P.S. Please, see also but 101230.

How to repeat:
Try to build MySQL 8.0.22 from source with option -DWITH_PROTOBUF=system and old Protobuf system library installed (e.g. version 3.2).

Suggested fix:
I expect that the build system would properly detect the installed Protobuf version on system level and raise an error, if the minimal required version is not available like this is done with other required libraries, instead of starting the build, which would inevitably fail with compilation errors. In short CMake should fail the build if the required minimum Protobuf version is not installed on system level.

Just to confirm that MySQL 8.0.22 builds without problem (i.e. without compilation errors) with system Protobuf 3.11.4 in otherwise the same environment, which confirms that 3.11.x is now the minimum required version.

The relevant lines form the log:

...
-- WITH_PROTOBUF=system
-- Found Protobuf: /usr/lib/libprotobuf.so;-lpthread (found version "3.11.4")
-- PROTOBUF_VERSION_NUMBER is #define GOOGLE_PROTOBUF_VERSION 3011004
-- PROTOBUF_INCLUDE_DIR /usr/include
-- PROTOBUF_LIBRARY /usr/lib/libprotobuf.so
-- PROTOBUF_LITE_LIBRARY /usr/lib/libprotobuf-lite.so
-- PROTOBUF_PROTOC_EXECUTABLE /usr/bin/protoc
-- Found CURL: /usr/lib/libcurl.so (found version "7.73.0")
...

Hi Mr. Sotirov,

Thank you for your bug report.

We have tried the same setup and finished with the same error.

Verified as reported.

Setting correct version.

Posted by developer:
 
I downloaded and built: https://github.com/protocolbuffers/protobuf/releases/tag/v3.2.0
(note, there is no 3.2.1 release)

wget https://github.com/protocolbuffers/protobuf/releases/download/v3.2.0/protobuf-cpp-3.2.0.ta...

and installed it to /tmp/proto320

Checking one of your missing symbols:
nm /tmp/proto320/lib/libprotoc.so.12  | grep _ZNK6google8protobuf8compiler13CodeGenerator11GenerateAllERKSt6vectorIPKNS0_14FileDescriptorESaIS6_EERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEPNS1_16GeneratorContextEPSG_
000000000009ba90 T _ZNK6google8protobuf8compiler13CodeGenerator11GenerateAllERKSt6vectorIPKNS0_14FileDescriptorESaIS6_EERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEPNS1_16GeneratorContextEPSG_
0000000000083c76 t _ZNK6google8protobuf8compiler13CodeGenerator11GenerateAllERKSt6vectorIPKNS0_14FileDescriptorESaIS6_EERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEPNS1_16GeneratorContextEPSG_.cold

To me it seems something is wrong with your system protobuf.

Thank you , Tor.

There is of course version 3.2.1 of Protobuf (see https://github.com/protocolbuffers/protobuf/releases/tag/v3.2.1), which I'm building like this:

autoreconf -vif
./configure --prefix=/usr \
            --libdir=/usr/lib \
            --docdir=/usr/doc/protobuf-3.2.1 \
            --enable-shared=yes \
            --enable-static=no \
            --enable-silent-rules \
            --with-zlib
make V=1 -j4 && make install

This gives me the following when searching for the first undefined symbol:

$ nm /usr/lib/libprotoc.so.12.0.0 | grep WriteStringMaybeAliased
         U _ZN6google8protobuf8internal14WireFormatLite23WriteStringMaybeAliasedEiRKSsPNS0_2io17CodedOutputStreamE

Why is that I cannot say, because I'm not that familiar with Protobuf library, but I definitely do not have this problem with Protobuf 3.11.2 built in the same way.

Wasn't this already verified? Please, check MySQL Verification Team's comment from 2020-10-21, which states "We have tried the same setup and finished with the same error." - how did they got to the same error as me?

nm /tmp/proto321/lib/libprotobuf.so  | grep WriteStringMaybeAliased
00000000000ead90 T _ZN6google8protobuf8internal14WireFormatLite23WriteStringMaybeAliasedEiRKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEPNS0_2io17CodedOutputStreamE
00000000000c8fde t _ZN6google8protobuf8internal14WireFormatLite23WriteStringMaybeAliasedEiRKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEPNS0_2io17CodedOutputStreamE.cold

Hi,

MySQL verification team has run with this problem with version 3.2.1. Since downgrading that library, the problems have gone.

the difference seems to be:

std::__cxx11::basic_string<char,<char> 
vs
std::basic_string<char, std::char_traits<char>

MySQl is built -std=c++14 which will use cxx11 strings

OK. That's an excellent clue. Thanks.

These are the flags for my compilation of 8.0.21:

-- CMAKE_CXX_FLAGS: -std=c++14 -fno-omit-frame-pointer -ffp-contract=off -ftls-model=initial-exec -O3 -march=i586 -mtune=i686 -fPIC -Wall -Wextra -Wformat-security -Wvla -Wundef -Woverloaded-virtual -Wcast-qual -Wimplicit-fallthrough=2 -Wstringop-truncation -Wlogical-op

And these are for 8.0.22:

-- CMAKE_CXX_FLAGS: -std=c++14 -fno-omit-frame-pointer -ffp-contract=off -ftls-model=initial-exec -O3 -march=i586 -mtune=i686 -fPIC -Wall -Wextra -Wformat-security -Wvla -Wundef -Woverloaded-virtual -Wcast-qual -Wimplicit-fallthrough=2 -Wstringop-truncation -Wsuggest-override -Wlogical-op

Clearly -std=c++14 is present in both, but I only encountered the reported problem when trying to build MySQL 8.0.22. There is a subtle difference (flag -Wsuggest-override added for 8.0.22), which I do not think is related.

I checked my build log for Protobuf 3.2.1 (with GCC 5.5.0) and it indicates that it was done with -std=c++11 flag, but it's not something I enforced. In the build for Protobuf 3.11.4 (with GCC 9.3.0) there is no -std flag, but C++14 is the default in GCC from 6.1 to 10 (see https://gcc.gnu.org/projects/cxx-status.html#cxx14). This rather explains why my build with Protobuf 3.11.4 is OK, but I'm still not sure how until MySQL 8.0.21 there was no problem when building with Protobuf 3.2.1. Anyway, I think it's clear now that the problem was into my build environment (Protobuf and MySQL built with different compilation flags), so please, close the bug.

P.S. I would however continue building MySQL with the same Protobuf version that is shipped in the source archive (i.e. 3.11.4).

Thanks for your feedback!

Closing bug as requested.

Thank you, Terje .......