Bug #68532 MySQL Client gives segmentation fault error when trying to connect
Submitted: 1 Mar 2013 3:58 Modified: 20 Jun 2014 0:28
Reporter: Tommy Peterson Email Updates:
Status: No Feedback Impact on me:
None 
Category:MySQL Server: Installing Severity:S2 (Serious)
Version:5.6.10 OS:Linux
Assigned to: Tor Didriksen CPU Architecture:Any
Tags: mysql client segmentation fault

[1 Mar 2013 3:58] Tommy Peterson
Description:
I installed the generic MySQL 5.6.10 Linux tar file to build for a RedHat x86_64 GNU/Linux server. The cmake/build/build install works fine. Starting and stopping MySQLd works fine. However, when I try to connect to the mysql client on the command line it appears to connect but terminates with SEGMENTATION FAULT error. I have restarted the Linux server and even reinstalled MySQL several times. Nothing changes. I can write a PHP script with PHP source build on the same Linux server to bind with this same MySQL build and it will connect just fine. So it is definitely the MySQL Client. I chose the tar file to build as I already have a MySQL on this same server and want to run this one separately. 

How to repeat:
Installed bison, m4, and libaio.

Set path:
PATH=$PATH:/opt/app/bison/bin:/opt/app/m4/bin:/opt/app/cmake/bin:/opt/app/libaio/bin
 
Downloaded and untar'd source code from Oracle. 

Made with the following:
/opt/app/cmake/bin/cmake . -DWITH_ARCHIVE_STORAGE_ENGINE=1 -DWITH_FEDERATED_STORAGE_ENGINE=1 -DMYSQL_DATADIR=/opt/app/mysql5610/data/ -DCMAKE_INSTALL_PREFIX=/opt/app/mysql5610 -DCURSES_LIBRARY=/opt/app/ncurses/lib/libncurses.a -DCURSES_INCLUDE_PATH=/opt/app/ncurses/include/ -DHAVE_LIBAIO_H=/opt/app/libaio/include/ -DINSTALL_LAYOUT=STANDALONE -DENABLED_PROFILING=ON -DMYSQL_MAINTAINER_MODE=OFF -DWITH_DEBUG=ON -DDEFAULT_CHARSET=utf8 -DENABLED_LOCAL_INFILE=TRUE -DWITH_ZLIB=bundled -DMYSQL_TCP_PORT=3309 -DMYSQL_UNIX_ADDR=/opt/app/mysql5610/mysql5610.sock

Issued make command

Issued make install command

Went into /opt/app/mysql5610/ and modified support-files/mysql.server by adding:
DATADIR=/opt/app/mysql5610/data/ and basedir=/opt/app/mysql5610

copied mysql.server to /opt/app/mysql5610/bin

Went into /opt/app/mysql5610/ and modified scripts/mysql_install_db by adding:
DATADIR=/opt/app/mysql5610/data/ and basedir=/opt/app/mysql5610

created the following my.cnf in /opt/app/mysql5610/
[client]
port=3309
socket=/opt/app/mysql5610/mysql5610.sock

[mysqld]
port=3309
socket=/opt/app/mysql5610/mysql5610.sock
pid-file= /opt/app/mysql5610/mysql5610.pid
general-log=/opt/app/mysql5610/mysql5610d.log

basedir="/opt/app/mysql5610"
datadir="/opt/app/mysql5610/data"

skip-external-locking

character-set-server=utf8
collation-server=utf8_general_ci

explicit_defaults_for_timestamp

ran /opt/app/mysql5610/scripts/mysql_install_db --defaults-file=/opt/app/mysql5610/my.cnf --port=3309 --user=admin --basedir=/opt/app/mysql5610 --datadir=/opt/app/mysql5610/data/ --pid-file=/opt/app/mysql5610/mysql5610.pid --socket=/opt/app/mysql5610/mysql5610.sock

started server /opt/app/mysql5610/bin/mysqld --defaults-file=/opt/app/mysql5610/my.cnf --port=3309 --user=admin --basedir=/opt/app/mysql5610 --datadir=/opt/app/mysql5610/data/ --pid-file=/opt/app/mysql5610/mysql5610.pid --socket=/opt/app/mysql5610/mysql5610.sock 

set root password /opt/app/mysql5610/bin/mysqladmin --defaults-file=/opt/app/mysql5610/my.cnf --port=3309  -u root password 'password*09';

checked the service was running with
ps aux | grep mysql

tried to start client with
/opt/app/mysql5610/bin/mysql --defaults-file=/opt/app/mysql5610/my.cnf --port=3309 -u root -p
Enter password: password*09

Get:
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 11
Server version: 5.6.10-debug Source distribution

Copyright (c) 2000, 2013, Oracle and/or its affiliates. All rights reserved.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Segmentation fault
sh-3.2$

Suggested fix:
Have no clue.
[1 Mar 2013 16:03] Tommy Peterson
Tried the following per an Oracle forum user's advice; didn't work:
"I think you may have hit on the problem of linking with the bundled libedit, but then running against a system libedit which is incompatible with it.

The fix is to edit cmd-line-utils/libedit/CMakeLists.txt thus: find the line

ADD_LIBRARY(edit ${LIBEDIT_SOURCES})

and change this to

ADD_CONVENIENCE_LIBRARY(edit ${LIBEDIT_SOURCES}) "
[6 Mar 2013 9:57] Erlend Dahl
Can you please send us

- full trace from cmake
- stacktrace from coredump (if you don't get a stacktrace please run the client in gdb)
- output from ldd /opt/app/mysql5610/bin/mysql

Thanks!
[7 Apr 2013 1:00] Bugs System
No feedback was provided for this bug for over a month, so it is
being suspended automatically. If you are able to provide the
information that was originally requested, please do so and change
the status of the bug back to "Open".
[24 Apr 2013 23:39] Deepti Kurup
We too had similar issues installing mysql-5.6.10. Even upgraded to 5.6.11 and still have same issue. Here is some details that might help re-open this case:

CMAKE OPTIONS:

cmake . -DWITH_ARCHIVE_STORAGE_ENGINE=1 -DWITH_FEDERATED_STORAGE_ENGINE=1 -DWITH_BLACKHOLE_STORAGE_ENGINE=1 -DMYSQL_DATADIR=$DASHOME/mysql-5.6.11/data/ -DCMAKE_INSTALL_PREFIX=$DASHOME/mysql-5.6.11 -DCURSES_LIBRARY=$DASHOME/lib/libncurses.a -DCURSES_INCLUDE_PATH=$DASHOME/include/ -DINSTALL_LAYOUT=STANDALONE -DENABLED_PROFILING=ON -DMYSQL_MAINTAINER_MODE=OFF -DWITH_DEBUG=OFF -DDEFAULT_CHARSET=utf8 -DDEFAULT_COLLATION=utf8_unicode_ci -DWITH_LIBEDIT=1 -DUSE_LIBEDIT_INTERFACE=1 -USE_NEW_READLINE_INTERFACE=0 -DENABLED_LOCAL_INFILE=TRUE -DWITH_ZLIB=bundled

=========================================================
Compilation warnings…. 

mysql-5.6.11/cmd-line-utils/libedit/terminal.c: In function ‘terminal_set’:
mysql-5.6.11/cmd-line-utils/libedit/terminal.c:893: warning: implicit declaration of function ‘tgetent’
mysql-5.6.11/cmd-line-utils/libedit/terminal.c:911: warning: implicit declaration of function ‘tgetflag’
mysql-5.6.11/cmd-line-utils/libedit/terminal.c:920: warning: implicit declaration of function ‘tgetnum’
mysql-5.6.11/cmd-line-utils/libedit/terminal.c:926: warning: implicit declaration of function ‘tgetstr’
mysql-5.6.11/cmd-line-utils/libedit/terminal.c:926: warning: cast to pointer from integer of different size
mysql-5.6.11/cmd-line-utils/libedit/terminal.c: In function ‘terminal_tputs’:
mysql-5.6.11/cmd-line-utils/libedit/terminal.c:1237: warning: implicit declaration of function ‘tputs’
mysql-5.6.11/cmd-line-utils/libedit/terminal.c: In function ‘terminal_echotc’:
mysql-5.6.11/cmd-line-utils/libedit/terminal.c:1539: warning: assignment makes pointer from integer without a cast

Try and run the mysql client with redirected input/reading an input file.. it does not have a problem.   Running it and trying to let it prompt you from the “mysql>” prompt and it core dumps.  I found the problem was with the ‘defines’ and mis-definition of some of the tgetstr,etc. functions as above.  Here is the core traceback for this… 

GNU gdb (GDB) Red Hat Enterprise Linux (7.0.1-42.el5)
Copyright (C) 2009 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from mysql-5.6.11/client/mysql...(no debugging symbols found)...done.
(gdb) r -u root -p --port=3306 --socket=/tmp/mysql.sock
Starting program: mysql-5.6.11/client/mysql -u root -p --port=3306 --socket=/tmp/mysql.sock
warning: no loadable sections found in added symbol-file system-supplied DSO at 0x2aaaaaaab000
[Thread debugging using libthread_db enabled]
[New Thread 0x40a00940 (LWP 21274)]
Enter password: [Thread 0x40a00940 (LWP 21274) exited]

Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 9
Server version: 5.6.11 Source distribution

Copyright (c) 2000, 2013, Oracle and/or its affiliates. All rights reserved.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Program received signal SIGSEGV, Segmentation fault.
0x00000000004a8bb4 in terminal_alloc ()
(gdb) where
#0  0x00000000004a8bb4 in terminal_alloc ()
#1  0x00000000004a9efd in terminal_set ()
#2  0x00000000004aa091 in terminal_init ()
#3  0x00000000004a0050 in el_init ()
#4  0x00000000004997ea in rl_initialize ()
#5  0x000000000049a7df in read_history ()
#6  0x0000000000413a46 in main ()
(gdb)

Then we added this to: terminal.c   on line 77 after the ‘#include “el.h”  …. since it is not able to include the definitions properly with the defines set….   This got us to the prompt and the moment we pressed <enter> after typing a command like “show databases;’ or “help” , it cored again in another place.

extern char * tgetstr (char *, char **);
extern char * tgoto (const char *, int, int);
extern int tgetent (char *, const char *);
extern int tgetflag (char *);
extern int tgetnum (char *);
extern int tputs (const char *, int, int (*)(int));

Compiled cleanly…. Then tested…  and got through the original problem.  Now it core dumps at a new issue. Gdb traceback follows..
make
Scanning dependencies of target edit
[  0%] Building C object cmd-line-utils/libedit/CMakeFiles/edit.dir/terminal.c.o
Linking C static library libedit.a

(gdb) r -u root -p --port=3306 --socket=/tmp/mysql.sock
Starting program: mysql-5.6.11/client/mysql -u root -p --port=3306 --socket=/tmp/mysql.sock
warning: no loadable sections found in added symbol-file system-supplied DSO at 0x2aaaaaaab000
[Thread debugging using libthread_db enabled]
[New Thread 0x40a00940 (LWP 17616)]
Enter password: [Thread 0x40a00940 (LWP 17616) exited]

Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 8
Server version: 5.6.11 Source distribution

Copyright (c) 2000, 2013, Oracle and/or its affiliates. All rights reserved.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql> help;                            # or any other command like show databases; ,etc… 

Program received signal SIGSEGV, Segmentation fault.
0x000000000047efc3 in my_uca_charcmp ()
(gdb) where
#0  0x000000000047efc3 in my_uca_charcmp ()
#1  0x000000000047f334 in my_wildcmp_uca_impl ()
#2  0x000000000047f3e9 in my_wildcmp_uca ()
#3  0x000000000040e9a6 in add_filtered_history(char const*) ()
#4  0x000000000041267b in read_and_execute(bool) ()
#5  0x0000000000413bc3 in main ()
[25 Apr 2013 6:48] Tor Didriksen
Deepti:
FindCurses.cmake obviously did not locate your curses library properly.

You should install the curses-devel (or whatever it's called) package
or alternatively run cmake with
-DCMAKE_PREFIX_PATH=$DASHOME

And please remove all your options for CURSES and LIBEDIT and READLINE
That should allow our cmake code to find the bundled libedit sources properly.
(5.6.11 cannot be compiled against your system libedit)

The problem with static/dynamic libedit is fixed in 5.6.11
(bug#68231)
[25 Apr 2013 14:17] Deepti Kurup
Tor,

We removed the items you suggested and ncurses has been compiled/installed in $DASHOME as before.  Here is our cmake line now:
    cmake . -DWITH_ARCHIVE_STORAGE_ENGINE=1 -DWITH_FEDERATED_STORAGE_ENGINE=1 -DWITH_BLACKHOLE_STORAGE_ENGINE=1 -DMYSQL_DATADIR=$DASHOME/mysql-5.6.11/data/ -DCMAKE_PREFIX_PATH=$DASHOME -DCMAKE_INSTALL_PREFIX=$DASHOME/mysql-5.6.11 -DINSTALL_LAYOUT=STANDALONE -DENABLED_PROFILING=ON -DMYSQL_MAINTAINER_MODE=OFF -DWITH_DEBUG=OFF -DDEFAULT_CHARSET=utf8 -DDEFAULT_COLLATION=utf8_unicode_ci -DENABLED_LOCAL_INFILE=TRUE -DWITH_ZLIB=bundled

The same warnings were generated again:
[  0%] Building C object cmd-line-utils/libedit/CMakeFiles/edit.dir/terminal.c.o
mysql-5.6.11/cmd-line-utils/libedit/terminal.c: In function ‘terminal_set’:
mysql-5.6.11/cmd-line-utils/libedit/terminal.c:901: warning: implicit declaration of function ‘tgetent’
mysql-5.6.11/cmd-line-utils/libedit/terminal.c:919: warning: implicit declaration of function ‘tgetflag’
mysql-5.6.11/cmd-line-utils/libedit/terminal.c:928: warning: implicit declaration of function ‘tgetnum’
mysql-5.6.11/cmd-line-utils/libedit/terminal.c:932: warning: implicit declaration of function ‘tgetstr’
mysql-5.6.11/cmd-line-utils/libedit/terminal.c:933: warning: passing argument 3 of ‘terminal_alloc’ makes pointer from integer without a cast
mysql-5.6.11/cmd-line-utils/libedit/terminal.c: In function ‘terminal_tputs’:

The mysql client abends like before with the same gdb backtrace.  I did more tests and found that the include/ncurses/term.h file is never included/read with the changes made as you suggested.  Ncurses is compiled/installed in $DASHOME  and the header files are in $DASHOME/include/ncurses.
[30 Apr 2013 17:48] Deepti Kurup
Tor,

Have you found anything else that we could test in this situation? If there is any other information we can provide to help resolve this, please let me know.
[2 May 2013 7:35] Tor Didriksen
Deepti

It seems you have to install the curses library where it belongs:
/usr/include
/usr/lib/ or /usr/lib64
[2 May 2013 16:29] Deepti Kurup
Tor,

Using ncurses under /usr/ is not an option for us. We are trying to build an environment that is not dependent on the default system libs as we have little control over upgrading/ updating them in future. I know, we managed to use the custom ncurses location for 5.5.27 version.

In your previous suggestions, you asked to remove the ncurses definition from the config file. Is there any other way we could hack it and force it to pick the custom ncurses location?
[14 May 2013 21:07] Deepti Kurup
Tor,

Is there any chance this will be investigated further. Do we go back and re-use 5.5.27 that happens to be working with custom libcurses location? Is there any chance of a new release with the fix anytime soon? Any help is appreciated.
[15 May 2013 8:07] Tor Didriksen
"happens to be working" seems appropriate.
I installed my own version of curses in /tmp/curses

with a little extra debug, this is what I get in 5.5 and 5.6

=== modified file 'cmake/readline.cmake'
--- cmake/readline.cmake        2013-02-26 05:45:28 +0000
+++ cmake/readline.cmake        2013-05-15 07:59:34 +0000
@@ -71,6 +71,11 @@
 MACRO (FIND_CURSES)
  FIND_PACKAGE(Curses) 
  MARK_AS_ADVANCED(CURSES_CURSES_H_PATH CURSES_FORM_LIBRARY CURSES_HAVE_CURSES_H)
+ MESSAGE(STATUS "CURSES_CURSES_H_PATH ${CURSES_CURSES_H_PATH}")
+ MESSAGE(STATUS "CURSES_FORM_LIBRARY ${CURSES_FORM_LIBRARY}")
+ MESSAGE(STATUS "CURSES_HAVE_CURSES_H ${CURSES_HAVE_CURSES_H}")
+ MESSAGE(STATUS "CURSES_INCLUDE_PATH ${CURSES_INCLUDE_PATH}")
+ MESSAGE(STATUS "CURSES_INCLUDE_DIR ${CURSES_INCLUDE_DIR}")
  IF(NOT CURSES_FOUND)

cmake .. -DCMAKE_PREFIX_PATH=/tmp/curses

-- Found Curses: /usr/lib64/libcurses.so  
-- CURSES_CURSES_H_PATH /usr/include
-- CURSES_FORM_LIBRARY /tmp/curses/lib/libform.a
-- CURSES_HAVE_CURSES_H /usr/include/curses.h
-- CURSES_INCLUDE_PATH /usr/include
-- CURSES_INCLUDE_DIR /usr/include

So FindCurses.cmake finds most of the stuff in /usr
except libform.a
[15 May 2013 8:12] Tor Didriksen
Your missing declarations belong to "term.h"

What "term.h" are you getting?

You can do
cd <build-dir>/cmd-line-utils/libedit
make terminal.i

and inspect the output from the pre-processor.
[15 May 2013 8:18] Tor Didriksen
The functionality of FindXXX.cmake varies a lot between cmake versions.
I guess I could replace FindCurses.cmake with my own 
FIND_LIBRARY and FIND_PATH code
(this is actually what I did for openssl in ssl.cmake)

It isn't particularly hard to do,
but it needs a lot of testing/verification on multiple platforms.
[16 May 2013 18:17] Deepti Kurup
Hi Tor,

we did the '

cd <build-dir>/cmd-line-utils/libedit
make terminal.i

and it was as we expected. It did not include term.h anywhere in it.
We also added the changes to the cmake/readline.cmake and it still generated the errors in the prototype when it compiled the code. I have attached the "terminal.c.i" for your reference.
[14 Jun 2013 15:36] Deepti Kurup
Tor,

We tried mysql version 5.6.12 and upgraded OS to RHEL 5.9 too.  The problem persists.

Program received signal SIGSEGV, Segmentation fault.
0x00000000004a8d54 in terminal_alloc ()
(gdb) where
#0  0x00000000004a8d54 in terminal_alloc ()
#1  0x00000000004aa09d in terminal_set ()
#2  0x00000000004aa231 in terminal_init ()
#3  0x00000000004a01f0 in el_init ()
#4  0x000000000049998a in rl_initialize ()
#5  0x000000000049a97f in read_history ()
#6  0x0000000000413b5d in main ()
[20 May 2014 0:28] MySQL Verification Team
Please try version 5.6.17. Thanks.
[20 Jun 2014 1:00] Bugs System
No feedback was provided for this bug for over a month, so it is
being suspended automatically. If you are able to provide the
information that was originally requested, please do so and change
the status of the bug back to "Open".