MySQL Bugs: #47657: MySQL 5.1.39: Crashed on installing system databases

Bug #47657	MySQL 5.1.39: Crashed on installing system databases
Submitted:	26 Sep 2009 6:22	Modified:	13 Oct 2016 13:20
Reporter:	Dinh Pham	Email Updates:
Status:	Can't repeat	Impact on me:	None
Category:	MySQL Server: Installing	Severity:	S1 (Critical)
Version:	5.1.39	OS:	Linux (Linux web2 2.6.18-92.1.18.el5.028stab060.8 #1 SMP Mon Feb 9 21:44:50 MSK 2009 x86_64 x86_64 x86_64 G)
Assigned to:		CPU Architecture:	Any
Tags:	crashed, linux, mysql_install_db, regression

Description:
I have tried to compile Mysql 5.1.39 from source (.zip file). Compiling was a success but I could not install system databases. A crash was reported:

[root@web2 pcdinh]# /usr/local/mysql/bin/mysql_install_db --general_log --defaults-file=/usr/local/mysql/etc/my.cnf --datadir=/usr/local/mysql/data --user=mysql
Installing MySQL system tables...

Installation of system tables failed!  Examine the logs in
/usr/local/mysql/data for more information.

You can try to start the mysqld daemon with:

    shell> /usr/local/mysql/bin/mysqld --skip-grant &

and use the command line tool /usr/local/mysql/bin/mysql
to connect to the mysql database and look at the grant tables:

    shell> /usr/local/mysql/bin/mysql -u root mysql
    mysql> show tables

Try 'mysqld --help' if you have problems with paths.  Using --log
gives you a log in /usr/local/mysql/data that may be helpful.

The latest information about MySQL is available on the web at
http://www.mysql.com/.  Please consult the MySQL manual section
'Problems running mysql_install_db', and the manual section that
describes problems on your OS.  Another information source are the
MySQL email archives available at http://lists.mysql.com/.

Please check all of the above before mailing us!  And remember, if
you do mail us, you MUST use the /usr/local/mysql/bin/mysqlbug script!

--------------------------------------------
db.err file content

090926 12:52:07 - mysqld got signal 11 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help diagnose
the problem, but since we have already crashed, something is definitely wrong
and this may fail.

key_buffer_size=402653184
read_buffer_size=2097152
max_used_connections=0
max_threads=400
threads_connected=1
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 2035687 K
bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

thd: 0xad62440
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0x41fa9f58 thread_stack 0x30000
[0x771c94]
[0x4be382]
[0x6396f0]
[0x76e6ef]
[0x71dbd4]
[0x71f698]
[0x71fb64]
[0x720fb1]
[0x547781]
[0x52e58c]
[0x542b6c]
[0x544768]
[0x545099]
[0x4d30bf]
[0x4d4748]
[0x4d61ce]
[0x6361d7]
[0x8053c9]
Trying to get some variables.
Some pointers may be invalid and cause the dump to abort...
thd->query at 0xad6d2b0 = INSERT INTO db SELECT * FROM tmp_db WHERE @had_db_table=0
thd->thread_id=1
thd->killed=NOT_KILLED
The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you find out what is causing the crash.
090926 13:14:10 - mysqld got signal 11 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help diagnose
the problem, but since we have already crashed, something is definitely wrong
and this may fail.

key_buffer_size=402653184
read_buffer_size=2097152
max_used_connections=0
max_threads=400
threads_connected=1
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 2035687 K
bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

thd: 0x3253840
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0x41e5df58 thread_stack 0x30000

 

How to repeat:
Compiling configuration

export MAKEOPTS="-j4"

CC="gcc" CFLAGS="-O3" CXX="gcc" CXXFLAGS="-O3 \
-felide-constructors -fno-exceptions -fno-rtti" \
./configure \
--prefix=/usr/local/mysql \
--enable-assembler \
--localstatedir=/usr/local/mysql/data \
--datadir=/usr/local/mysql/data \
--libexecdir=/usr/local/mysql/bin \
--libdir=/usr/local/mysql/lib \
--with-server-suffix=-max \
--enable-local-infile \
--enable-thread-safe-client \
--enable-shared \
--with-fast-mutexes \
--with-libwrap \
--with-zlib-dir=bundled \
--with-readline \
--with-plugins=partition,myisam \
--with-plugins=innobase \
--without-embedded-server \
--with-charset=utf8 \
--with-collation=utf8_general_ci \
--with-extra-charsets=all \
--with-unix-socket-path=/tmp/mysql.sock \
--with-client-ldflags=-all-static \
--with-mysqld-ldflags=-all-static \
--with-big-tables \
--with-atomic-ops=smp \
--with-max-indexes=128 \
--with-mysqld-user=mysql

make
make install

Thank you for the problem report. Please, send the resolved stack trace and the results of:

file mysqld
free

Linux commands. I want to check if your mysqld binary is 32-bit and how much free memory do you have.

Send also your my.cnf file content.

Here is the link that I used to download source code http://dev.mysql.com/get/Downloads/MySQL-5.1/mysql-5.1.39.zip/from/pick

Some more information

Free memory
-----------

             total       used       free     shared    buffers     cached
Mem:       8171444    7992848     178596          0      79416    4069436
-/+ buffers/cache:    3843996    4327448
Swap:      2031608     430560    1601048

CPU
-----------
4 cores

processor       : 3

GCC 
-----------
Using built-in specs.
Target: x86_64-redhat-linux
Configured with: ../configure --prefix=/usr --mandir=/usr/share/man --infodir=/usr/share/info --enable-shared --enable-threads=posix --enable-checking=release --with-system-zlib --enable-__cxa_atexit --disable-libunwind-exceptions --enable-libgcj-multifile --enable-languages=c,c++,objc,obj-c++,java,fortran,ada --enable-java-awt=gtk --disable-dssi --enable-plugin --with-java-home=/usr/lib/jvm/java-1.4.2-gcj-1.4.2.0/jre --with-cpu=generic --host=x86_64-redhat-linux
Thread model: posix
gcc version 4.1.2 20080704 (Red Hat 4.1.2-44)
vendor_id       : AuthenticAMD
cpu family      : 15
model           : 33
model name      : Dual Core AMD Opteron(tm) Processor 290
stepping        : 2
cpu MHz         : 2791.094
cache size      : 1024 KB
physical id     : 1
siblings        : 2
core id         : 1
cpu cores       : 2
fpu             : yes
fpu_exception   : yes
cpuid level     : 1
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt lm 3dnowext 3dnow pni lahf_lm cmp_legacy
bogomips        : 5581.20
TLB size        : 1024 4K pages
clflush size    : 64
cache_alignment : 64
address sizes   : 40 bits physical, 48 bits virtual
power management: ts fid vid ttp

file mysqld
--------------------
mysqld: ELF 64-bit LSB executable, AMD x86-64, version 1 (SYSV), for GNU/Linux 2.6.9, statically linked, for GNU/Linux 2.6.9, not stripped

ulimit -a
--------------------
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
max nice                        (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 66048
max locked memory       (kbytes, -l) 32
max memory size         (kbytes, -m) unlimited
open files                      (-n) 1024
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
max rt priority                 (-r) 0
stack size              (kbytes, -s) 10240
cpu time               (seconds, -t) unlimited
max user processes              (-u) 66048
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

my.cnf
--------------------------
[client]
port            = 3306
socket          = /tmp/mysql.sock

[mysqld]
port            = 3306
socket          = /tmp/mysql.sock
skip-locking
skip-name-resolve

default-storage-engine=MyISAM

default-character-set=utf8
character-set-server=utf8
collation_server=utf8_general_ci

key_buffer_size = 384M
key_buffer = 384M
max_allowed_packet = 8M
table_open_cache = 512
sort_buffer_size = 2M
read_buffer_size = 2M
read_rnd_buffer_size = 8M
join_buffer_size = 5M
bulk_insert_buffer_size = 32M
thread_concurrency = 8
thread_stack = 192K
thread_cache_size=8

query_cache_limit=2M

query_cache_size=64M

query_cache_type=1

connect_timeout=10
max_connections = 400
max_connect_errors = 10000
wait_timeout=600 

concurrent_insert=2

max_heap_table_size = 50M
net_buffer_length = 512K

tmp_table_size = 64M
tmpdir = /dev/shm

myisam_sort_buffer_size = 64M
myisam_recover = force,backup

ft_min_word_len=1
ft_max_word_len=30

log-error=/usr/local/mysql/data/db1.err

slow_query_log=1
slow_query_log_file=/usr/local/mysql/data/Clip-slow.log
long_query_time=1

expire_logs_days=1

log-bin=mysql-bin

server-id       = 1

[mysqldump]
quick
max_allowed_packet = 16M

[mysql]
no-auto-rehash

[myisamchk]
key_buffer_size = 256M
sort_buffer_size = 256M
read_buffer = 2M
write_buffer = 2M
# Full text search
ft_min_word_len=1
ft_max_word_len=30

[mysqlhotcopy]
interactive-timeout

[mysqld_safe]
open_files_limit = 16344

Hi Valeriy,

> Please, send the resolved stack trace 

How could I do that? Do I need to re-compile MySQL with --with-debug to get the trace?

Trace 1
--------------------
0x771c94 ha_myisammrg::extra_opt(ha_extra_function, unsigned long) + 2
0x4be382 check_single_table_access(THD*, unsigned long, TABLE_LIST*, bool) + 62
0x6396f0 Event_scheduler::execute_top(Event_queue_element_for_exec*) + 372
0x790db7 add_collation + 426
0x766e9b merge_index + 154
0x51183e st_table::restore_column_maps_after_mark_index() + 98
0x5123c7 table_check_intact(st_table*, unsigned int, st_table_field_w_type const*) + 35
0x5c0ee0 mysql_upgrade_db(THD*, st_mysql_lex_string*) + 816
0x4d3999 __static_initialization_and_destruction_0(int, int) + 17661
0x4d4748 __static_initialization_and_destruction_0(int, int) + 21164
0x4d61ce __static_initialization_and_destruction_0(int, int) + 27954
0x6361d7 Table_triggers_list::create_trigger(THD*, TABLE_LIST*, String*) + 445
0x8053c9 __parse_one_specmb + 857

Trace 2
-----------------------

0x771c94 ha_myisammrg::extra_opt(ha_extra_function, unsigned long) + 2
0x4be382 check_single_table_access(THD*, unsigned long, TABLE_LIST*, bool) + 62
0x6396f0 Event_scheduler::execute_top(Event_queue_element_for_exec*) + 372
0x5bb95b ha_release_savepoint(THD*, st_savepoint*) + 23
0x5bbb05 ha_savepoint(THD*, st_savepoint*) + 185
0x61ec9c mysqld_help(THD*, char const*) + 512
0x50fa6c TABLE_LIST::setup_underlying(THD*) + 532
0x511813 st_table::restore_column_maps_after_mark_index() + 55
0x5123c7 table_check_intact(st_table*, unsigned int, st_table_field_w_type const*) + 35
0x5126f6 table_check_intact(st_table*, unsigned int, st_table_field_w_type const*) + 850
0x62c793 sp_head::execute_function(THD*, Item**, unsigned int, Field*) + 987
0x4beeac log_slow_statement(THD*) + 182
0x4c318b mysql_execute_command(THD*) + 2325
0x7d4e00 sin + 848
0x4001b9 _start + 41

Trace 3 (MySQL built with --with-debug
-------------------------------

Error log
---------
thd: 0x1ba22d10
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0x40b17f50 thread_stack 0x30000
[0x794ef9]
[0x4b3980]
[0x657e00]
[0x50371f]
[0x49eae7]
[0x4c9c26]
[0x654d97]
[0x824789]
Trying to get some variables.
Some pointers may be invalid and cause the dump to abort...
thd->query at 0x1ba2dde0 = INSERT INTO tmp_db VALUES ('%','test','','Y','Y','Y','Y','Y','Y','N','Y','Y','Y','Y','Y','Y','Y','Y','N','N','Y','Y')
thd->thread_id=1
thd->killed=KILL_CONNECTION
The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you find out what is causing the crash.

Trace 
-----

0x794ef9 my_print_stacktrace + 41
0x4b3980 handle_segfault + 574
0x657e00 __restore_rt + 0
0x50371f close_temporary_tables(THD*) + 517
0x49eae7 THD::cleanup() + 199
0x4c9c26 handle_bootstrap + 970
0x654d97 start_thread + 199
0x824789 clone + 121

I just made a clean re-compilation and installation (without debug mode). Here is the latest trace on the crash

0x771c94 my_write_core + 20
0x4be382 handle_segfault + 818
0x6396f0 __restore_rt + 0
0x5bb95b tablename_to_filename(char const*, char*, unsigned int) + 27
0x5bbb05 build_table_filename(char*, unsigned long, char const*, char const*, char const*, unsigned int) + 261
0x61ec9c Table_triggers_list::check_n_load(THD*, char const*, char const*, st_table*, bool) + 76
0x50fa6c open_unireg_entry(THD*, st_table*, TABLE_LIST*, char const*, char*, unsigned int, st_mem_root*, unsigned int) + 1196
0x511813 open_table(THD*, TABLE_LIST*, st_mem_root*, bool*, unsigned int) + 2067
0x5123c7 open_tables(THD*, TABLE_LIST**, unsigned int*, unsigned int) + 1639
0x5126f6 open_and_lock_tables_derived(THD*, TABLE_LIST*, bool) + 70
0x62c793 plugin_init(int*, char**, int) + 1715
0x4beeac init_server_components() + 524
0x4c318b main + 1771
0x7d4e00 ____strtoll_l_internal + 528
0x4001b9 _start + 41

Thank you for the feedback.

Verified as described using mysql-5.1-bugteam tree: compile with options provided, run mysql_install_db, get crash.

Bug is not repeatable with version 5.0

Seems like all options are needed as I can not repeat this if remove one.

Also problem is not repeatable in my environment if LD_LIBRARY_PATH contains directory with 5.0 libraries.

Hi,

Is there any update on this issue?

Dinh

I had the same problem with MySQL 5.1.37

No longer reproducible with current versions of MySQL.