MySQL Bugs: #40680: Falcon: linking fails on Mac OS X

Bug #40680	Falcon: linking fails on Mac OS X
Submitted:	12 Nov 2008 21:47	Modified:	19 Mar 2009 11:44
Reporter:	Hakan Küçükyılmaz	Email Updates:
Status:	Can't repeat	Impact on me:	None
Category:	MySQL Server: Falcon storage engine	Severity:	S3 (Non-critical)
Version:	6.0.8	OS:	MacOS (10.4)
Assigned to:	Hakan Küçükyılmaz	CPU Architecture:	Any
Tags:	F_PLATFORM

Description:
ccache gcc -g -DSAFE_MUTEX -DSAFEMALLOC -Wimplicit -Wreturn-type -Wswitch -Wtrigraphs -Wcomment -W -Wchar-subscripts -Wformat -Wparentheses -Wsign-compare -Wwrite-strings -Wunused-function -Wunused-label -Wunused-value -Wunused-variable -Wreorder -Wctor-dtor-privacy -Wnon-virtual-dtor -felide-constructors -fno-exceptions -fno-rtti -mtune=pentium-m -DUNIV_MUST_NOT_INLINE -DEXTRA_DEBUG -DFORCE_INIT_OF_VARS -DSAFEMALLOC -DPEDANTIC_SAFEMALLOC -DSAFE_MUTEX -fno-implicit-templates -fno-exceptions -fno-rtti -D_P1003_1B_VISIBLE -DSIGNAL_WITH_VIO_CLOSE -DSIGNALS_DONT_BREAK_READ -DIGNORE_SIGHUP_SIGQUIT -DDONT_DECLARE_CXA_PURE_VIRTUAL -o mysqld sql_lex.o sql_handler.o sql_partition.o item.o item_sum.o item_buff.o item_func.o item_cmpfunc.o item_strfunc.o item_timefunc.o thr_malloc.o item_create.o item_subselect.o item_row.o item_geofunc.o item_xmlfunc.o field.o strfunc.o key.o sql_class.o sql_list.o net_serv.o protocol.o sql_state.o lock.o my_lock.o sql_string.o sql_manager.o sql_map.o mysqld.o password.o hash_filo.o hostname.o sql_connect.o scheduler.o sql_parse.o set_var.o sql_yacc.o sql_base.o table.o sql_select.o sql_insert.o sql_profile.o sql_prepare.o sql_error.o sql_locale.o sql_update.o sql_delete.o uniques.o sql_do.o procedure.o sql_test.o log.o init.o derror.o sql_acl.o unireg.o des_key_file.o log_event.o rpl_record.o log_event_old.o rpl_record_old.o discover.o time.o opt_range.o opt_sum.o records.o filesort.o handler.o ha_partition.o debug_sync.o sql_db.o sql_table.o sql_rename.o sql_crypt.o sql_load.o mf_iocache.o field_conv.o sql_show.o sql_udf.o sql_analyse.o sql_cache.o slave.o sql_repl.o rpl_filter.o rpl_tblmap.o rpl_utility.o rpl_injector.o rpl_rli.o rpl_mi.o rpl_reporting.o sql_union.o sql_derived.o sql_client.o repl_failsafe.o sql_olap.o sql_view.o gstream.o spatial.o sql_help.o sql_cursor.o tztime.o my_decimal.o sp_head.o sp_pcontext.o sp_rcontext.o sp.o sp_cache.o parse_file.o sql_trigger.o event_scheduler.o event_data_objects.o event_queue.o event_db_repository.o events.o sql_plugin.o sql_binlog.o sql_builtin.o sql_tablespace.o partition_info.o sql_servers.o sql_audit.o sha2.o ddl_blocker.o si_objects.o si_logs.o event_parse_data.o mdl.o transaction.o rpl_handler.o mini_client_errors.o pack.o client.o my_time.o my_user.o -Wl,-bind_at_load  ./.libs/libndb.a ../storage/myisam/libmyisam.a ../storage/archive/libarchive.a ../storage/blackhole/libblackhole.a ../storage/csv/libcsv.a ../storage/falcon/libfalcon.a ../storage/federated/libfederated.a ../storage/heap/libheap.a ../storage/innobase/libinnobase.a ../storage/maria/libmaria.a ../storage/myisammrg/libmyisammrg.a ../vio/libvio.a ../mysys/libmysys.a ../dbug/libdbug.a ../regex/libregex.a ../strings/libmystrings.a -lz -lstdc++ -ldl ../extra/yassl/src/.libs/libyassl.a ../extra/yassl/taocrypt/src/.libs/libtaocrypt.a backup/.libs/libbackup.a -lm  
/usr/libexec/gcc/i686-apple-darwin8/4.0.1/ld: warning empty table of contents: ./.libs/libndb.a (can't load from it)
/usr/libexec/gcc/i686-apple-darwin8/4.0.1/ld: warning multiple definitions of symbol ___cxa_pure_virtual
../mysys/libmysys.a(my_new.o) definition of ___cxa_pure_virtual in section (__TEXT,__text)
/usr/lib/gcc/i686-apple-darwin8/4.0.1/libstdc++.dylib(single module) definition of ___cxa_pure_virtual
/usr/libexec/gcc/i686-apple-darwin8/4.0.1/ld: Undefined symbols:
__Unwind_Resume
collect2: ld returned 1 exit status
make[3]: *** [mysqld] Error 1
make[2]: *** [all-recursive] Error 1
make[1]: *** [all] Error 2
make: *** [all-recursive] Error 1

How to repeat:
Use latest mysql-6.0-falcon tree from bzr and do
    BUILD/compile-pentium-debug-max-no-ndb

Suggested fix:
Add appropriate -fexception when building with Falcon

A simple binary search revealed that this regression happened in the mysql-6.0-falcon tree between revisions 2875 and 2876:

=== modified file 'storage/falcon/plug.in'
--- storage/falcon/plug.in	2008-08-20 13:27:29 +0000
+++ storage/falcon/plug.in	2008-09-23 09:08:59 +0000
@@ -66,33 +66,40 @@
   AC_CACHE_CHECK([compiler flag to enable implicit templates],
                  falcon_cxx_enable_implicit_templates,[
     CXXFLAGS="$ac_save_CXXFLAGS -fimplicit-templates"
-    AC_TRY_LINK([],
-      [int i = 0;
-       i++;],
-      falcon_cxx_enable_implicit_templates="-fimplicit-templates",
-      falcon_cxx_enable_implicit_templates=""
+    AC_LINK_IFELSE([AC_LANG_PROGRAM()],
+      falcon_cxx_enable_implicit_templates="-fimplicit-templates"
     )
   ])
 
+  # -lstdc++ really shouldn't be here, and is only required while we continue
+  # to use gcc to link C++ code for the static linking hack.  Once we stop
+  # doing that it should be removed from here, and the CXXLDFLAG setting below
+  # will do the right thing for g++.
   AC_CACHE_CHECK([compiler flag to enable exceptions],
                  falcon_cxx_enable_exceptions,[
-    CXXFLAGS="$ac_save_CXXFLAGS -fexceptions -lstdc++"
-    AC_TRY_LINK([],
-      [int i = 0;
-       i++;],
-      falcon_cxx_enable_exceptions="-fexceptions -lstdc++",
-      falcon_cxx_enable_exceptions=""
-    )
+    m4_foreach([cxxflag], [[-fexceptions -lstdc++], [-features=except]],[
+      CXXFLAGS="$ac_save_CXXFLAGS cxxflag"
+      AC_LINK_IFELSE([AC_LANG_PROGRAM()],
+        falcon_cxx_enable_exceptions="cxxflag"
+      )
+    ])
+  ])
+
+  AC_CACHE_CHECK([linker flag to enable STL],
+                 falcon_cxx_enable_stl,[
+    m4_foreach([cxxflag], [-lstdc++],[
+      CXXFLAGS="$ac_save_CXXFLAGS cxxflag"
+      AC_LINK_IFELSE([AC_LANG_PROGRAM()],
+        falcon_cxx_enable_stl="cxxflag"
+      )
+    ])
   ])
 
   AC_CACHE_CHECK([compiler flag to silence offsetof warnings],
                  falcon_cxx_no_invalid_offsetof,[
     CXXFLAGS="$ac_save_CXXFLAGS -Wno-invalid-offsetof"
-    AC_TRY_LINK([],
-      [int i = 0;
-       i++;],
-      falcon_cxx_no_invalid_offsetof="-Wno-invalid-offsetof",
-      falcon_cxx_no_invalid_offsetof=""
+    AC_LINK_IFELSE([AC_LANG_PROGRAM()],
+      falcon_cxx_no_invalid_offsetof="-Wno-invalid-offsetof"
     )
   ])
 
@@ -100,7 +107,7 @@
   AC_LANG_POP([C++])
 
   # Falcon uses exceptions and STL.
-  CXXLDFLAGS="$CXXLDFLAGS $falcon_cxx_enable_exceptions"
+  CXXLDFLAGS="$CXXLDFLAGS $falcon_cxx_enable_stl"
   FALCON_CXXFLAGS="$falcon_cxx_no_invalid_offsetof $falcon_cxx_enable_implicit_templates $falcon_cxx_enable_exceptions"
   case "$with_debug" in
     yes)  FALCON_CXXFLAGS="$FALCON_CXXFLAGS -D_DEBUG" ;;

config.log of revision 2876 (not working)

Attachment: config.log.gz (application/x-tar, text), 42.99 KiB.

config.log of revision 2875 (working)

Attachment: config.log.gz (application/x-tar, text), 42.64 KiB.

Failure to include -fexceptions is due to this lines

configure:52696: ccache gcc -o conftest -g -DSAFE_MUTEX -DSAFEMALLOC -Wimplicit -Wreturn-type -Wswitch -Wtrigraphs -Wcomment -W -Wchar-subscripts -Wformat -Wparentheses -Wsign-compare -Wwrite-strings -Wunused-function -Wunused-label -Wunused-value -Wunused-variable -Wreorder -Wctor-dtor-privacy -Wnon-virtual-dtor -felide-constructors -fno-exceptions -fno-rtti -mtune=pentium-m -DUNIV_MUST_NOT_INLINE -DEXTRA_DEBUG -DFORCE_INIT_OF_VARS  -DSAFEMALLOC -DPEDANTIC_SAFEMALLOC -DSAFE_MUTEX   -fno-implicit-templates -fno-exceptions -fno-rtti -D_P1003_1B_VISIBLE -DSIGNAL_WITH_VIO_CLOSE -DSIGNALS_DONT_BREAK_READ -DIGNORE_SIGHUP_SIGQUIT  -DDONT_DECLARE_CXA_PURE_VIRTUAL -fexceptions -lstdc++   conftest.cpp -lm  >&5
conftest.cpp:154:1: warning: "HAVE_PTHREAD_YIELD_NP" redefined
conftest.cpp:153:1: warning: this is the location of the previous definition
configure:52703: $? = 0
configure:52746: ccache gcc -o conftest -g -DSAFE_MUTEX -DSAFEMALLOC -Wimplicit -Wreturn-type -Wswitch -Wtrigraphs -Wcomment -W -Wchar-subscripts -Wformat -Wparentheses -Wsign-compare -Wwrite-strings -Wunused-function -Wunused-label -Wunused-value -Wunused-variable -Wreorder -Wctor-dtor-privacy -Wnon-virtual-dtor -felide-constructors -fno-exceptions -fno-rtti -mtune=pentium-m -DUNIV_MUST_NOT_INLINE -DEXTRA_DEBUG -DFORCE_INIT_OF_VARS  -DSAFEMALLOC -DPEDANTIC_SAFEMALLOC -DSAFE_MUTEX   -fno-implicit-templates -fno-exceptions -fno-rtti -D_P1003_1B_VISIBLE -DSIGNAL_WITH_VIO_CLOSE -DSIGNALS_DONT_BREAK_READ -DIGNORE_SIGHUP_SIGQUIT  -DDONT_DECLARE_CXA_PURE_VIRTUAL -features=except   conftest.cpp -lm  >&5
cc1plus: error: unrecognized command line option "-features=except"

This change is required to compile Falcon using the Sun Studio compiler.

The problem originates from the BUILD/* scripts using -fno-exceptions, a relic from earlier in the 6.0 process where we could get away with disabling exceptions and avoiding a runtime dependancy on libstdc++.  Now that Falcon requires exceptions, we can no longer get away with this hack, and therefore the -fno-exceptions flag is actively harmful.

This was masked previously as -fexceptions was added CXXLDFLAGS which overrides the default, however this is the wrong place to add a compiler flag and causes problems on platforms where the flag for enabling exceptions in the compiler is not supported by the linker - it only happens to work where gcc is used both to compile and link.

Removing it from the BUILD/* scripts should solve the problem, but needs testing, especially on platforms where the compiler may default to enabling exceptions across the server, which is incompatible in certain places (yassl comes to mind).

Just some minor comments:

1. It is actually not the two lines taken from config.log (see two comments above) that make this fail. These two lines correctly detects the correct and needed flags to compile with support for C++ exceptions. But these are only added to the compiler stage, not to the linking stage.

2. The reason the C++ exception flags are not added to the linker stage is the following change in the patch:

   # Falcon uses exceptions and STL.
-  CXXLDFLAGS="$CXXLDFLAGS $falcon_cxx_enable_exceptions"
+  CXXLDFLAGS="$CXXLDFLAGS $falcon_cxx_enable_stl"

I agree with Jonathan that it is probably not correct to add the C++ exception flags directly to the CXXLDFLAGS (although I think this change was not needed in order to compile Falcon with Sun Studio compiler?)

Would a possible solution be to introduce a FALCON_CXXLDFLAGS variable similar to the one we have for FALCON_CXXFLAGS?

Olav, Can this be closed now that the changes for Bug#39419 are done?

Kevin, bug#39419 does not fix this problem. The fix for bug#39419 does no changes to which options that are provided when compiling/linking a program, it just checks that whether the actually provided compiler options will result in a binary that supports C++ options.

We should keep this bug open until we have a fix for it.

Information provided by Olav;

This has only been seen/reported by Hakan this far. I asked Lars-Erik to try to reproduce it on his Mac but he does not see the problem. I have gotten log files from running configure from both Hakan and Lars-Erik and they are almost identical. I suspect this has something to do with which compiler version Hakan is using.

Still, Hakan is right about this being a regression. The change that caused this was done by Jonathan Perkins some time ago when the build team adjusted Falcon's configure to adhere to the way they configured for Sun Studio compiler. I think the change that Jonathan did was correct but unfortunately broke this on at least some Mac OS versions.

Fails to link in this way, too:

BUILD/autorun.sh

./configure \
  --disable-shared \
  --enable-assembler \
  --enable-local-infile \
  --enable-shared \
  --enable-thread-safe-client \
  --libexecdir=/usr/local/mysql/bin \
  --localstatedir=/usr/local/mysql/data \
  --with-big-tables \
  --with-client-ldflags='-static' \
  --with-comment='MySQL-Community-Server' \
  --with-extra-charsets=all \
  --with-falcon \
  --with-innodb \
  --with-partition \
  --prefix=/usr/local/mysql \
  --with-readline \
  --with-server-suffix='' \
  --with-ssl \
  --with-libevent \
  --with-zlib-dir=bundled

time make -j2

/usr/libexec/gcc/powerpc-apple-darwin8/4.0.1/ld: warning empty table of contents: ./.libs/libndb.a (can't load from it)
/usr/libexec/gcc/powerpc-apple-darwin8/4.0.1/ld: warning multiple definitions of symbol ___cxa_pure_virtual
../mysys/libmysys.a(my_new.o) definition of ___cxa_pure_virtual in section (__TEXT,__text)
/usr/lib/gcc/powerpc-apple-darwin8/4.0.1/libstdc++.dylib(single module) definition of ___cxa_pure_virtual
/usr/libexec/gcc/powerpc-apple-darwin8/4.0.1/ld: Undefined symbols:
__Unwind_Resume
collect2: ld returned 1 exit status
make[3]: *** [mysqld] Error 1
make[2]: *** [all-recursive] Error 1
make[1]: *** [all] Error 2
make: *** [all-recursive] Error 1

real    129m25.235s
user    84m16.040s
sys     22m39.741s

This way I could compile MySQL/Falcon with latest mysql-6.0-falcon sources:

BUILD/autorun.sh
CCACHE_TEST=$(ccache -V 2> /dev/null)
if [ "x$CCACHE_TEST" != "x" ]; then
    CCACHE=$(which ccache)
else
    CCACHE=''
fi

export CC="$CCACHE gcc"
export CXX="$CCACHE g++"

export CFLAGS="-g \
  -O3 \
  -fno-omit-frame-pointer \
  -Wunused-function \
  -Wunused-label \
  -Wunused-value \
  -Wunused-variable \
  -Wimplicit \
  -Wreturn-type \
  -Wswitch \
  -Wtrigraphs \
  -Wcomment \
  -W \
  -Wchar-subscripts \
  -Wformat \
  -Wparentheses \
  -Wsign-compare \
  -Wwrite-strings \
  -Wuninitialized \
  -Wunused \
  -DFORCE_INIT_OF_VARS"

export CXXFLAGS="-g \
  -O3 \
  -fno-omit-frame-pointer \
  -Wunused-function \
  -Wunused-label \
  -Wunused-value \
  -Wunused-variable \
  -Wimplicit \
  -Wreturn-type \
  -Wswitch \
  -Wtrigraphs \
  -Wcomment \
  -W \
  -Wchar-subscripts \
  -Wformat \
  -Wparentheses \
  -Wsign-compare \
  -Wwrite-strings \
  -Wctor-dtor-privacy \
  -Wnon-virtual-dtor \
  -Wuninitialized \
  -DFORCE_INIT_OF_VARS"

#
# We need shared libraries for DBT2's client and driver program.
#
export CONFIGURATION="--enable-assembler \
  --enable-local-infile \
  --enable-thread-safe-client \
  --libexecdir=/usr/local/mysql/bin \
  --localstatedir=/usr/local/mysql/data \
  --with-big-tables \
  --with-comment='MySQL-Community-Server' \
  --with-extra-charsets=all \
  --with-plugins=max-no-ndb \
  --prefix=/usr/local/mysql \
  --with-readline \
  --with-ssl \
  --with-libevent \
  --with-zlib-dir=bundled"

./configure $CONFIGURATION
make -j2

Hakan, What was the resolution of this?

Can't repeat with latest Falcon from mysql-6.0-falcon bzr tree anymore. Falcon builds fine with
    BUILD/compile-pentium-debug-max-no-ndb