MySQL Bugs: #60844: [mysql] 5.5.11-1 segmentation fault in mysql_install

Bug #60844	[mysql] 5.5.11-1 segmentation fault in mysql_install_db script
Submitted:	12 Apr 2011 18:06	Modified:	13 May 2011 14:03
Reporter:	Jeremy Dawson	Email Updates:
Status:	Can't repeat	Impact on me:	None
Category:	MySQL Server: Installing	Severity:	S1 (Critical)
Version:	5.5.11-1	OS:	Linux
Assigned to:		CPU Architecture:	Any
Tags:	archlinux

Description:
This is a fresh install besides the fact that I have tried multiple times, when I originally received this error I did a fresh os install of archlinux.

# I am getting a segmentation fault when installing mysql during the mysql_install_db script.
Here's the latest example when compiling from source:
[root@serverass mysql]# scripts/mysql_install_db --user=mysql
Installing MySQL system tables...
110412  9:21:09 - mysqld got signal 11 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help diagnose
the problem, but since we have already crashed, something is definitely wrong
and this may fail.

key_buffer_size=8388608
read_buffer_size=131072
max_used_connections=0
max_threads=151
thread_count=1
connection_count=0
It is possible that mysqld could use up to 
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 338483 K
bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0x1863300
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
scripts/mysql_install_db: line 428: 18160 Done                    { echo "use mysql;"; cat $create_system_tables $fill_system_tables; }
     18161                       | eval "$filter_cmd_line"
     18162 Segmentation fault      | $mysqld_install_cmd_line > /dev/null

Installation of system tables failed!  Examine the logs in
./data for more information.

When using the binary package from archlinux I modified the mysql_install_db file to echo out all its variables which were:
[2011-04-09 15:25] Running 'pacman -U --logfile=mysql_install.log mysql-5.5.11-1-x86_64.pkg.tar.xz'
[2011-04-09 15:25] arg is:
[2011-04-09 15:25] arg is:--basedir=/usr
[2011-04-09 15:25] arg is:--server-id=1
[2011-04-09 15:25] we made it here because basedir exists. basedir is:/usr
[2011-04-09 15:25] binder is:/usr/bin
[2011-04-09 15:25] extra_binder is:/usr/bin
[2011-04-09 15:25] pkgdatadir is:/usr/share/mysql
[2011-04-09 15:25] scriptdir is:/usr/scripts
[2011-04-09 15:25] fill_help_tables is:/usr/share/mysql/fill_help_tables.sql
[2011-04-09 15:25] create_system_tables is:/usr/share/mysql/mysql_system_tables.sql
[2011-04-09 15:25] fill_system_tables is:/usr/share/mysql/mysql_system_tables_data.sql
[2011-04-09 15:25] args after user is: --user=mysql
[2011-04-09 15:25] Now testing for cross_bootstrap...
[2011-04-09 15:25] filter_cmd_line is:cat
[2011-04-09 15:25] mysqld_bootstrap is:/usr/bin/mysqld
[2011-04-09 15:25] mysqld_install_cmd_line is:/usr/bin/mysqld --lc-messages-dir=/usr/share/mysql/english/.. --bootstrap --basedir=/usr --datadir=/var/lib/mysql --log-warnings=0 --loose-skip-innodb --loose-skip-ndbcluster --user=mysql --max_allowed_packet=8M --default-storage-engine=myisam --net_buffer_length=16K

How to repeat:
# using source
Followed steps in INSTALL-SOURCE section 2.9.2. Installing MySQL from a Standard Source Distribution
# Preconfiguration setup
shell> groupadd mysql
shell> useradd -r -g mysql mysql
# Beginning of source-build specific instructions
shell> tar zxvf mysql-VERSION.tar.gz
shell> cd mysql-VERSION
shell> cmake .
shell> make
shell> make install
# End of source-build specific instructions
# Postinstallation setup
shell> cd /usr/local/mysql
shell> chown -R mysql .
shell> chgrp -R mysql .
shell> scripts/mysql_install_db --user=mysql

# using binary package on archlinux
pacman -S mysql

I wasn't sure if this was an address to a stack trace:
Thread pointer: 0x1863300

So I went through those steps as well (from here: http://dev.mysql.com/doc/refman/5.5/en/using-stack-trace.html):
# /usr/local/mysql/bin/resolve_stack_dump -s /tmp/mysqld.sym -n /tmp/mystack.trace
0x1863300 _end + 9365552

Please, check if the same problem happens with generic Linux .tar.gz binaries available from dev.mysql.com.

So I downloaded the binary - mysql-5.5.11-linux2.6-x86_64.tar.gz

Went through the steps for setup to the mysql_install_db step:

shell> groupadd mysql
shell> useradd -r -g mysql mysql
shell> cd /usr/local
shell> tar zxvf /path/to/mysql-VERSION-OS.tar.gz
shell> ln -s full-path-to-mysql-VERSION-OS mysql
shell> cd mysql
shell> chown -R mysql .
shell> chgrp -R mysql .
shell> scripts/mysql_install_db --user=mysql

Now I am getting an error for a missing file:

# scripts/mysql_install_db --user=mysql
Installing MySQL system tables...
./bin/mysqld: error while loading shared libraries: libaio.so.1: cannot open shared object file: No such file or directory

# Installed the missing library
[root@serverass mysql]# pacman -Ss libaio
community/libaio 0.3.109-5
    The Linux-native asynchronous I/O facility (aio) library

Now its working!

[root@serverass mysql]# scripts/mysql_install_db --user=mysql
Installing MySQL system tables...
OK
Filling help tables...
OK

....

# 

I am not certain why the binary from archlinux didn't provide the error for this library nor did the compiled version with no custom flags.

Now that I got the error message its easier to fix.

Related bug: http://bugs.mysql.com/bug.php?id=60544

Can you please help figure out why this missing library may not have been in either of the above examples?

Problem with libaio (our binaries require it) is know, and you correctly identified it as Bug #60544.

I am not sure how ArchLinux binaries are built. But can you, please, send the exact sequence of commands you executed on your system to build from source?

The fact that our binaries work means that the problem is either in build environment difference or related to exact build commands used.

I did confirm that the archlinux binary and the built from source copy still fails on mysql_install_db as well.

These are my commands for building from source:
 groupadd mysql
 useradd -r -g mysql mysql
 tar -xzvf mysql-5.5.11.tar.gz
 cd mysql-5.5.11
 cmake .
 make
 make install
 cd /usr/local/mysql
 chown -R mysql .
 chgrp -R mysql .
 scripts/mysql_install_db --user=mysql

These are the options passed to the MySQL package in Arch Linux:

  CFLAGS="-fPIC -march=x86-64 -mtune=generic -O2 -pipe -fno-strict-aliasing -DBIG_JOINS=1 -fomit-frame-pointer" \
  CXXFLAGS="-fPIC -march=x86-64 -mtune=generic -O2 -pipe -fno-strict-aliasing -DBIG_JOINS=1 -felide-constructors -fno-rtti" \

  cmake . \
    -DCMAKE_BUILD_TYPE=Release \
    -DCMAKE_INSTALL_PREFIX=/usr \
    -DMYSQL_DATADIR=/var/lib/mysql \
    -DSYSCONFDIR=/etc/mysql \
    -DINSTALL_INFODIR=share/mysql/docs \
    -DINSTALL_MANDIR=share/man \
    -DINSTALL_PLUGINDIR=/usr/lib/mysql/plugin \
    -DINSTALL_SCRIPTDIR=bin \
    -DINSTALL_INCLUDEDIR=include/mysql \
    -DINSTALL_DOCREADMEDIR=share/mysql \
    -DINSTALL_SUPPORTFILESDIR=share/mysql \
    -DINSTALL_MYSQLSHAREDIR=share/mysql \
    -DINSTALL_SHAREDIR=share/mysql \
    -DWITH_READLINE=ON \
    -DWITH_ZLIB=system \
    -DWITH_SSL=system \
    -DWITH_LIBWRAP=ON \
    -DDEFAULT_CHARSET=utf8 \
    -DDEFAULT_COLLATION=utf8_general_ci \
    -DWITH_EXTRA_CHARSETS=complex \
    -DWITH_EMBEDDED_SERVER=ON \
    -DMYSQL_UNIX_ADDR=/var/run/mysqld/mysqld.sock \
    -DENABLED_LOCAL_INFILE=ON \
    -DWITH_PARTITION_STORAGE_ENGINE=1 \
    -DWITHOUT_EXAMPLE_STORAGE_ENGINE=1 \
    -DWITHOUT_ARCHIVE_STORAGE_ENGINE=1 \
    -DWITHOUT_BLACKHOLE_STORAGE_ENGINE=1 \
    -DWITHOUT_FEDERATED_STORAGE_ENGINE=1 \
    -DWITH_INNOBASE_STORAGE_ENGINE=1

So it appears a new version of mysql in now available. I have removed all other traces of previous versions and installed this mysql 5.5.12-1, the error is no longer there, its able to setup the tables and everything. I am not sure if this was a planned fix or not but the new version appears to be working as intended.

Not repeatable with 5.5.12.