Bug #68231 Segmentation Fault, mysql client
Submitted: 31 Jan 2013 7:22 Modified: 14 Feb 2013 18:34
Reporter: daryl xian Email Updates:
Status: Closed Impact on me:
None 
Category:MySQL Server: Command-line Clients Severity:S1 (Critical)
Version:5.6.10 GA / source build OS:Linux (opensuse 12.2)
Assigned to: Tor Didriksen CPU Architecture:Any

[31 Jan 2013 7:22] daryl xian
Description:
buidling 5.6.9-rc from src

	mysql --version
		mysql  Ver 14.14 Distrib 5.6.9-rc, for Linux (x86_64) using  EditLine wrapper

on linux/64

	uname -a
		Linux test 3.4.11-2.16-desktop #1 SMP PREEMPT Wed Sep 26 17:05:00 UTC 2012 (259fc87) x86_64 x86_64 x86_64 GNU/Linux

with

	gcc version 4.7.2 20130108 [gcc-4_7-branch revision 195012] (SUSE Linux)

mysqld is up & running, and accessible, e.g.,

	mysqlshow
		+---------------------------+
		|         Databases         |
		+---------------------------+
		| information_schema        |
		| TEST_db                   |
		| mysql                     |
		| performance_schema        |
		| test                      |
		+---------------------------+

mysql client can connect

	mysql
		Welcome to the MySQL monitor.  Commands end with ; or \g.
		Your MySQL connection id is 13
		Server version: 5.6.9-rc-log Source distribution

		Copyright (c) 2000, 2012, Oracle and/or its affiliates. All rights reserved.

		Oracle is a registered trademark of Oracle Corporation and/or its
		affiliates. Other names may be trademarks of their respective
		owners.

		Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

		mysql>

but any/all commands @ prompt cause a segfault, e.g.,

		mysql> help
			For information about MySQL products and services, visit:
			...
			nowarning (\w) Don't show warnings after every statement.

			For server side help, type 'help contents'

			Segmentation fault

@ syslog

Jan 30 22:25:45 test kernel: [609793.927100] mysql[21986]: segfault at 0 ip 000000000047e9eb sp 00007fff1d393d90 error 4 in mysql[400000+2b6000]

How to repeat:
build from src
start server
exec mysql
enter any command/query
[31 Jan 2013 7:23] daryl xian
gdb trace & valgrind output @ segfault

Attachment: mysql-gdb-valgrind.txt (text/plain), 86.57 KiB.

[31 Jan 2013 9:01] Tor Didriksen
Looking at the first valgrind warning below.
 - you have line numbers for libedit, but not for mysql?
 - comparing with my version of 5.6.9-release, the line number do not match
To me it looks like libedit crashes when trying to read your edit history.

How did you build (cmake command)
What version of libedit are you using, the one that comes with mysql?

		--26830-- REDIR: 0x637d6e0 (wcslen) redirected to 0x4c2f960 (wcslen)
		--26830-- REDIR: 0xffffffffff600400 (???) redirected to 0x3806c73d (???)
		--26830-- REDIR: 0x63194d0 (setenv) redirected to 0x4c2fac0 (setenv)
		==26830== Conditional jump or move depends on uninitialised value(s)
		==26830==    at 0x637CF1C: wcscmp (wcscmp.S:439)
		==26830==    by 0x505DD42: map_set_editor (map.c:1085)
		==26830==    by 0x505B3DC: el_wset (el.c:207)
		==26830==    by 0x506FA8A: el_set (eln.c:133)
		==26830==    by 0x506C26E: rl_initialize (readline.c:331)
		==26830==    by 0x506CFD9: read_history (readline.c:1308)
		==26830==    by 0x40B652: ??? (in /usr/local/mysql/bin/mysql)
		==26830==    by 0x6303454: (below main) (libc-start.c:226)
		==26830==  Uninitialised value was created by a heap allocation
		==26830==    at 0x4C2C26B: malloc (in /usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so)
		==26830==    by 0x4C2C51F: realloc (in /usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so)
		==26830==    by 0x505E524: ct_conv_buff_resize (chartype.c:66)
		==26830==    by 0x505E60E: __ct_decode_string (chartype.c:119)
		==26830==    by 0x506FA78: el_set (eln.c:133)
		==26830==    by 0x506C26E: rl_initialize (readline.c:331)
		==26830==    by 0x506CFD9: read_history (readline.c:1308)
		==26830==    by 0x40B652: ??? (in /usr/local/mysql/bin/mysql)
		==26830==    by 0x6303454: (below main) (libc-start.c:226)
[31 Jan 2013 14:38] daryl xian
> How did you build (cmake command)

cmake .. -DCMAKE_INSTALL_PREFIX=/usr/local/mysql \
-DWITH_DEBUG=0 -DENABLE_DEBUG_SYNC=0 \
-DBUILD_SHARED_LIBS=1 \
-DINSTALL_LAYOUT=STANDALONE \
-DINSTALL_LIBDIR=lib64 \
-DWITH_SSL=bundled \
-DCMAKE_C_FLAGS="-O2 -march=amdfam10 -mtune=amdfam10 -D_FORTIFY_SOURCE=2 -fstack-protector -funwind-tables -fasynchronous-unwind-tables-fmessage-length=0" \
-DCMAKE_CXX_FLAGS="-O2 -march=amdfam10 -mtune=amdfam10 -D_FORTIFY_SOURCE=2 -fstack-protector -funwind-tables -fasynchronous-unwind-tables -fmessage-length=0" \
-DDEFAULT_CHARSET=utf8 -DDEFAULT_COLLATION=utf8_unicode_ci -DWITH_EXTRA_CHARSETS=all \
-DENABLE_DOWNLOADS=0 -DWITH_UNIT_TESTS=0 \
-DWITH_EMBEDDED_SERVER=0 \
-DWITH_INNOBASE_STORAGE_ENGINE=1 \
-DWITH_PARTITION_STORAGE_ENGINE=1 \
-DWITH_PERFSCHEMA_STORAGE_ENGINE=1 \
-DWITH_INNODB_MEMCACHED=1 \
-DWITH_ARCHIVE_STORAGE_ENGINE=0 \
-DWITH_BLACKHOLE_STORAGE_ENGINE=0 \
-DWITH_EXAMPLE_STORAGE_ENGINE=0 \
-DWITH_FEDERATED_STORAGE_ENGINE=0 \
-DWITH_NDBCLUSTER_STORAGE_ENGINE=0

> What version of libedit are you using, the one that comes with mysql?

I didn't set libedit path/type specifically.  It apparently defaulted to:

	cmake .. -LAH | grep -i edit
		// Use bundled libedit
		WITH_LIBEDIT:BOOL=ON

where

	ldd `which mysql` | grep libedit
		libedit.so => /usr/lib64/libedit.so (0x00007ff9498d7000)
	rpm -q --whatprovides /usr/lib64/libedit.so
		libedit-devel-3.0.snap20110802-21.5.x86_64
[31 Jan 2013 14:52] daryl xian
fyi,

if I attempt to force use of READLINE as alternative (to libedit?) in the build, adding:

  -DWITH_READLINE=1 \

to cmake command line, @ `cmake ...`, I get:

...
CMake Warning:
  Manually-specified variables were not used by the project:

    WITH_READLINE
[31 Jan 2013 16:51] daryl xian
removing the documented, but rejected,

	-DWITH_READLINE=1 \

finding in mysql/config.h.cmake,

	#cmakedefine USE_LIBEDIT_INTERFACE 1
	#cmakedefine USE_NEW_READLINE_INTERFACE 1

(1) adding to command line,

	-DUSE_LIBEDIT_INTERFACE=0 \
	-DUSE_NEW_READLINE_INTERFACE 1 \

`cmake` is OK, `make` fails

	make
		...
		[ 75%] Built target mysql_client_test
		[ 75%] Building CXX object client/CMakeFiles/mysql.dir/mysql.cc.o
		/usr/local/src/mysql/client/mysql.cc: In function ‘void initialize_readline(char*)’:
		/usr/local/src/mysql/client/mysql.cc:2697:38: error: ‘rl_completion_func_t’ was not declared in this scope
		/usr/local/src/mysql/client/mysql.cc:2697:59: error: expected primary-expression before ‘)’ token
		/usr/local/src/mysql/client/mysql.cc:2698:56: error: invalid conversion from ‘char* (*)(const char*, int)’ to ‘int (*)(const char*, int)’ [-fpermissive]
		/usr/local/src/mysql/client/mysql.cc:2700:73: error: invalid conversion from ‘int (*)(int, int)’ to ‘int (*)(const char*, int)’ [-fpermissive]
		In file included from /usr/local/src/mysql/client/mysql.cc:100:0:
		/usr/local/src/mysql/cmd-line-utils/libedit/readline/readline.h:194:7: error:   initializing argument 2 of ‘int rl_add_defun(const char*, int (*)(const char*, int), int)’ [-fpermissive]
		/usr/local/src/mysql/client/mysql.cc: In function ‘void tee_write(FILE*, const char*, size_t, int)’:
		/usr/local/src/mysql/client/mysql.cc:5158:34: warning: ignoring return value of ‘size_t fwrite(const void*, size_t, size_t, FILE*)’, declared with attribute warn_unused_result [-Wunused-result]
		/usr/local/src/mysql/client/mysql.cc:5160:39: warning: ignoring return value of ‘size_t fwrite(const void*, size_t, size_t, FILE*)’, declared with attribute warn_unused_result [-Wunused-result]
		make[2]: *** [client/CMakeFiles/mysql.dir/mysql.cc.o] Error 1
		make[1]: *** [client/CMakeFiles/mysql.dir/all] Error 2
		make: *** [all] Error 2

(2) mod'ing command line to,

	-DUSE_LIBEDIT_INTERFACE=1 \
	-DUSE_NEW_READLINE_INTERFACE 0 \

`cmake` is OK, `make` is OK

daemon starts,

	systemctl restart mysqld-custom.service
	systemctl status  mysqld-custom.service
		mysqld-custom.service - MySQL database server
		          Loaded: loaded (/etc/systemd/system/mysqld-custom.service; enabled)
		          Active: active (running) since Thu, 31 Jan 2013 08:49:15 -0800; 7s ago
		         Process: 5143 ExecStartPost=/usr/local/libexec/mysqld-wait-ready $MAINPID (code=exited, status=0/SUCCESS)
		        Main PID: 5142 (mysqld_safe)
		          CGroup: name=systemd:/system/mysqld-custom.service
		                  ├ 5142 /bin/sh /usr/local/mysql/bin/mysqld_safe --defaults-file=/usr/local/etc/mysql/my.cnf
		                  └ 6072 /usr/local/mysql/bin/mysqld --defaults-file=/usr/local/etc/mysql/my.cnf --basedir=/usr/...

		Jan 31 08:49:13 test mysqld_safe[5142]: 130131 08:49:13 mysqld_safe Logging to '/var/log/mysql/mysql-err.log'.
		Jan 31 08:49:13 test mysqld_safe[5142]: 130131 08:49:13 mysqld_safe Starting mysqld daemon with database...mysql

	mysqld -V
		mysqld  Ver 5.6.9-rc for Linux on x86_64 (Source distribution)

mysql client segfaults, as reported,

	mysql
	Welcome to the MySQL monitor.  Commands end with ; or \g.
	Your MySQL connection id is 2
	Server version: 5.6.9-rc-log Source distribution

	Copyright (c) 2000, 2012, Oracle and/or its affiliates. All rights reserved.

	Oracle is a registered trademark of Oracle Corporation and/or its
	affiliates. Other names may be trademarks of their respective
	owners.

	Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

	mysql> help
		For information about MySQL products and services, visit:
		...
		Segmentation fault
[1 Feb 2013 4:52] daryl xian
upgrading to a local build of *latest* libedit from http://www.thrysoee.dk/editline/libedit-20121213-3.0.tar.gz,

	pkg-config --libs --cflags libedit
		-I/usr/local/include -I/usr/local/include/editline  -L/usr/local/lib64 -ledit -ltinfo

config of mysql,

	cmake .. \
	...
	-DWITH_LIBEDIT=1 \
	-DUSE_LIBEDIT_INTERFACE=1 \
	-DUSE_NEW_READLINE_INTERFACE=0 \
	...
	make
	make install

verifying,

	ldd /usr/local/mysql/bin/mysql | grep libedit
	        libedit.so => /usr/local/lib64/libedit.so (0x00007fae13d58000)

unfortunately

	mysql
	mysql> help
	...
	Segmentation fault
[1 Feb 2013 5:25] daryl xian
valrgind with local libedit seems better behaved.  output attached.
[1 Feb 2013 5:27] daryl xian
valgrind output with local libedit

Attachment: mysql+local_libedit-valgrind.txt (text/plain), 77.04 KiB.

[1 Feb 2013 6:33] daryl xian
configuring with

	cmake .. \
	...
	-DWITH_LIBEDIT=1 \
	-DUSE_LIBEDIT_INTERFACE=1 \
	-DUSE_NEW_READLINE_INTERFACE=0 \
	...

replacing

	...
	-DDEFAULT_CHARSET=utf8 -DDEFAULT_COLLATION=utf8_unicode_ci \
	...

with

	...
	-DDEFAULT_CHARSET=latin1 -DDEFAULT_COLLATION=latin1_swedish_ci \
	...

results in a `mysql` that no longer segfaults.  `mysql` functions as normal.

however,

	cmake .. \
	...
	-DWITH_LIBEDIT=1 \
	-DUSE_LIBEDIT_INTERFACE=0 \
	-DUSE_NEW_READLINE_INTERFACE=1 \
	-DDEFAULT_CHARSET=latin1 -DDEFAULT_COLLATION=latin1_swedish_ci \
	...

still fails

	make
		...
		[ 70%] Building CXX object client/CMakeFiles/mysql.dir/mysql.cc.o
		/usr/local/src/mysql/client/mysql.cc: In function ‘void initialize_readline(char*)’:
		/usr/local/src/mysql/client/mysql.cc:2697:38: error: ‘rl_completion_func_t’ was not declared in this scope
		/usr/local/src/mysql/client/mysql.cc:2697:59: error: expected primary-expression before ‘)’ token
		/usr/local/src/mysql/client/mysql.cc:2698:56: error: invalid conversion from ‘char* (*)(const char*, int)’ to ‘int (*)(const char*, int)’ [-fpermissive]
		/usr/local/src/mysql/client/mysql.cc:2700:73: error: invalid conversion from ‘int (*)(int, int)’ to ‘int (*)(const char*, int)’ [-fpermissive]
		In file included from /usr/local/src/mysql/client/mysql.cc:100:0:
		/usr/local/src/mysql/cmd-line-utils/libedit/readline/readline.h:194:7: error:   initializing argument 2 of ‘int rl_add_defun(const char*, int (*)(const char*, int), int)’ [-fpermissive]
		/usr/local/src/mysql/client/mysql.cc: In function ‘void tee_write(FILE*, const char*, size_t, int)’:
		/usr/local/src/mysql/client/mysql.cc:5158:34: warning: ignoring return value of ‘size_t fwrite(const void*, size_t, size_t, FILE*)’, declared with attribute warn_unused_result [-Wunused-result]
		/usr/local/src/mysql/client/mysql.cc:5160:39: warning: ignoring return value of ‘size_t fwrite(const void*, size_t, size_t, FILE*)’, declared with attribute warn_unused_result [-Wunused-result]
		make[2]: *** [client/CMakeFiles/mysql.dir/mysql.cc.o] Error 1
		make[1]: *** [client/CMakeFiles/mysql.dir/all] Error 2
		make: *** [all] Error 2
[1 Feb 2013 12:13] Tor Didriksen
This is the reason I guess:
-DBUILD_SHARED_LIBS=1 \

When I do that, I get this in my build dir:
$ldd ./client/mysql
        linux-vdso.so.1 =>  (0x00007fffdaf67000)
        libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f74eebf2000)
        libedit.so => /export-b/home/didrik/repo/5.6.9-rc-release/bin-foo/cmd-line-utils/libedit/libedit.so (0x00007f74ee9b1000)
        librt.so.1 => /lib64/librt.so.1 (0x00007f74ee7a9000)
        libdl.so.2 => /lib64/libdl.so.2 (0x00007f74ee5a5000)
        libncurses.so.5 => /lib64/libncurses.so.5 (0x00007f74ee380000)
        libtinfo.so.5 => /lib64/libtinfo.so.5 (0x00007f74ee157000)
        libstdc++.so.6 => /lib64/libstdc++.so.6 (0x00007f74ede54000)
        libm.so.6 => /lib64/libm.so.6 (0x00007f74edb58000)
        libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007f74ed943000)
        libc.so.6 => /lib64/libc.so.6 (0x00007f74ed58c000)
        /lib64/ld-linux-x86-64.so.2 (0x00007f74eee38000)

In cmd-line-utils/libedit/CMakeLists.txt there is:
ADD_LIBRARY(edit ${LIBEDIT_SOURCES})

it should have been:
ADD_LIBRARY(edit STATIC ${LIBEDIT_SOURCES})

Can you try it out in your environment?
[1 Feb 2013 15:11] daryl xian
Sry, I'm not clear what you're asking me to try.

The segfault goes away if I change cmake configure

	-	-DDEFAULT_CHARSET=utf8 -DDEFAULT_COLLATION=utf8_unicode_ci \
	+	-DDEFAULT_CHARSET=latin1 -DDEFAULT_COLLATION=latin1_swedish_ci \

Are you suggesting I modify cmd-line-utils/libedit/CMakeLists.txt

	-	ADD_LIBRARY(edit ${LIBEDIT_SOURCES})
	+	ADD_LIBRARY(edit STATIC ${LIBEDIT_SOURCES})

?
[1 Feb 2013 16:05] Tor Didriksen
yes, try linking statically, with the libedit that comes with the mysql sources.
so: modify our CMakeLists.txt
[1 Feb 2013 16:32] daryl xian
mod'ing

	vi cmd-line-utils/libedit/CMakeLists.txt

		-	ADD_LIBRARY(edit ${LIBEDIT_SOURCES})
		+	ADD_LIBRARY(edit STATIC ${LIBEDIT_SOURCES})

config with

	cmake .. \
	...
	-DWITH_LIBEDIT=1 \
	-DUSE_LIBEDIT_INTERFACE=1 \
	-DUSE_NEW_READLINE_INTERFACE=0 \
	-DDEFAULT_CHARSET=latin1 -DDEFAULT_COLLATION=latin1_swedish_ci \
	...
	make
	make install

checking

	ldd /usr/local/mysql/bin/mysql | grep libedit
		(empty)

	mysql
	mysql> help
		...
		OK (NO segfault)

reversing, to check,

	vi cmd-line-utils/libedit/CMakeLists.txt

		-	ADD_LIBRARY(edit STATIC ${LIBEDIT_SOURCES})
		+	ADD_LIBRARY(edit ${LIBEDIT_SOURCES})

	cmake .. \
	...
	make
	make install
	ldd /usr/local/mysql/bin/mysql | grep libedit
		libedit.so => /usr/lib64/libedit.so (0x00007f19f7aa3000)
	mysql
	mysql> help
		...
		OK (NO segfault)

back to STATIC,

	vi cmd-line-utils/libedit/CMakeLists.txt

		-	ADD_LIBRARY(edit ${LIBEDIT_SOURCES})
		+	ADD_LIBRARY(edit STATIC ${LIBEDIT_SOURCES})

mod'ing CHARSET @ config with

	cmake .. \
	...
	-DWITH_LIBEDIT=1 \
	-DUSE_LIBEDIT_INTERFACE=1 \
	-DUSE_NEW_READLINE_INTERFACE=0 \
	-DDEFAULT_CHARSET=utf8 -DDEFAULT_COLLATION=utf8_unicode_ci \
	...
	make
	make install

	ldd /usr/local/mysql/bin/mysql | grep libedit
		(empty)

	mysql
	mysql> help
		...
		OK (NO segfault)

SUMMARY:

with

	-DWITH_LIBEDIT=1 \
	-DUSE_LIBEDIT_INTERFACE=1 \
	-DUSE_NEW_READLINE_INTERFACE=0 \

`cmake ...` & `make` OK. result,

	(1) system, dynamic libedit, charset/collate latin1/latin1_swedish_ci  -> NO segfault
	(2) system, dynamic libedit, charset/collate utf8/utf8_unicode_ci      -> SEGFAULT
	(3) bundled STATIC libedit,  charset/collate latin1/latin1_swedish_ci  -> NO segfault
	(4) bundled STATIC libedit,  charset/collate utf8/utf8_unicode_ci      -> NO segfault

with

	-DWITH_LIBEDIT=1 \
	-DUSE_LIBEDIT_INTERFACE=0 \
	-DUSE_NEW_READLINE_INTERFACE=1 \

`make` fails in all cases.
[5 Feb 2013 15:36] daryl xian
bumping Version from 

5.6.9-rc / source build

to 

5.6.10 GA / source build
[14 Feb 2013 18:34] Paul DuBois
Noted in 5.6.11, 5.7.1 changelogs.

If mysql is built with the bundled libedit library, the library is
built as static code, to avoid linking to a different dynamic version
at runtime. Dynamic linking could result in use of a different,
incompatible version and a segmentation fault.