Bug #43741 Mysql server core dump when running tests 'execution_constants'
Submitted: 19 Mar 2009 8:08 Modified: 3 Dec 2009 9:21
Reporter: Sunanda Menon Email Updates:
Status: Verified Impact on me:
None 
Category:MySQL Server: General Severity:S2 (Serious)
Version:5.0.67, 5.0.77,5.0.81,5.0.86 OS:Other (OpenSolaris X86 (nvb110) and now with nvb119)
Assigned to: CPU Architecture:Any
Tags: core, crash, Solaris x86
Triage: Triaged: D5 (Feature request)

[19 Mar 2009 8:08] Sunanda Menon
Description:
The test execution_constants.test generates a crash of the server on X86 machine.

How to repeat:
To reproduce the problem launch the following command :

cd /usr/mysql/mysql-test
/usr/mysql/mysql-test/mysql-test-run  --vardir=/var/tmp/var --verbose t/execution_constants.test

and a core file is generated in /var/tmp/var/master-data/core

<suffrene:mysql-test>pstack /var/tmp/var/master-data/core
core '/var/tmp/var/master-data/core' of 2182:   /usr/mysql/5.0/bin/mysqld --no-defaults --basedir=/usr/mysql/5.0 --cha
-----------------  lwp# 1 / thread# 1  --------------------
 bf8c1687 __pollsys (8045a70, 2, 0, 0) + 7
 bf86ce01 pselect  (9, 8045c20, 0, 0, 0, 0) + 199
 bf86d1d6 select   (9, 8045c20, 0, 0, 0, 89254ac) + 78
 08242b1c handle_connections_sockets (0) + 324
 0824223f main     (1e, 80466b4, 8046730, 81852f0) + 111f
 0818538d _start   (1e, 804693c, 8046956, 8046964, 804697d, 80469b6) + 7d
-----------------  lwp# 2 / thread# 2  --------------------
 bf8c0fc7 __sigtimedwait (bfe3ef84, 0, 0, bf8ae9c7, 8921c88) + 7
 bf8ae9dd sigwait  (bfe3ef84, bf94f000, bfe3ef5c, bf89871b) + 22
 bf89872c __posix_sigwait (bfe3ef84, bfe3ef78, 0, 14) + 34
 0823f503 signal_hand (0, bf94f000, bfe3efec, bf8bcd1e) + 1c7
 bf8bcd56 _thrp_setup (bfdf0200) + 7e
 bf8bcfe0 _lwp_start (bfdf0200, 0, 0, bf8bcd1e, 0, 0)
-----------------  lwp# 3 / thread# 3  --------------------
 0829e606 __1cXfind_field_in_table_ref6FpnDTHD_pnKTABLE_LIST_pkcI555ppnEItem_bbpIbp3_pnFField__ (8b90028, 8bfc068, 8ceced8, 7, 8ceced8, 0) + 9e2
 0829f0c2 __1cUfind_field_in_tables6FpnDTHD_pnKItem_ident_pnKTABLE_LIST_5ppnEItem_nbBfind_item_error_report_type_bb_pnFField__ (8b90028, 8cecee0, 8bfc068, 0, 8ced02c, 4) + 96a
 0819885d __1cKItem_fieldKfix_fields6MpnDTHD_ppnEItem__b_ (8cecee0, 8b90028, 8ced02c, 81b34f5) + 29d
 081b3575 __1cJItem_funcKfix_fields6MpnDTHD_ppnEItem__b_ (8cecfd8, 8b90028, 8ced4e8, 81b34f5) + 91
 081b3575 __1cJItem_funcKfix_fields6MpnDTHD_ppnEItem__b_ (8ced468, 8b90028, 8ced580, 81d008e) + 91
 081d00ba __1cMItem_func_ifKfix_fields6MpnDTHD_ppnEItem__b_ (8ced468, 8b90028, 8ced580, 81b34f5) + 3a
 081b3575 __1cJItem_funcKfix_fields6MpnDTHD_ppnEItem__b_ (8ced4f8, 8b90028, 8ced610, 81d008e) + 91
 081d00ba __1cMItem_func_ifKfix_fields6MpnDTHD_ppnEItem__b_ (8ced4f8, 8b90028, 8ced610, 81b34f5) + 3a
 081b3575 __1cJItem_funcKfix_fields6MpnDTHD_ppnEItem__b_ (8ced588, 8b90028, 8ced6a0, 81d008e) + 91
 081d00ba __1cMItem_func_ifKfix_fields6MpnDTHD_ppnEItem__b_ (8ced588, 8b90028, 8ced6a0, 81b34f5) + 3a
 081b3575 __1cJItem_funcKfix_fields6MpnDTHD_ppnEItem__b_ (8ced618, 8b90028, 8ced730, 81d008e) + 91
.........
 081d00ba __1cMItem_func_ifKfix_fields6MpnDTHD_ppnEItem__b_ (8cd97a8, 8b90028, 8cd98c0, 81b34f5) + 3a
 081b3575 __1cJItem_funcKfix_fields6MpnDTHD_ppnEItem__b_ (8cd9838, 8b90028, 8cd9950, 81d008e) + 91
 081d00ba __1cMItem_func_ifKfix_fields6MpnDTHD_ppnEItem__b_ (8cd9838, 8b90028, 8cd9950, 81b34f5) + 3a
 081b3575 __1cJItem_funcKfix_fields6MpnDTHD_ppnEItem__b_ (8cd98c8, 8b90028, 8cd9964, 81d008e) + 91
 081d00ba __1cMItem_func_ifKfix_fields6MpnDTHD_ppnEItem__b_ (8cd98c8, 8b90028, 8cd9964, 82a0da6) + 3a
 082a0f5a __1cMsetup_fields6FpnDTHD_ppnEItem_rnEList4n0B___bp5b_b_ (8b90028, 0, 8b91150, 1, 0, 0) + 1c2
 082ebd51 __1cMmysql_update6FpnDTHD_pnKTABLE_LIST_rnEList4nEItem___5pn0C_IpnIst_order_XnPenum_duplicates_b_i_ (8b90028, 8bfc068, 8b90f78, 8b91150, 8cd9e20, 0) + 475
 0825c8a1 __1cVmysql_execute_command6FpnDTHD__i_ (8b90028, 8b90028, 8cbe050, 3ca5) + 2c9
 0825aa8b __1cQdispatch_command6FnTenum_server_command_pnDTHD_pcI_b_ (3, 8b90028, 8bb4019, 3ca6) + 2b9b
 08256f96 handle_one_connection (8b90028, bf94f000, bfb7efec, bf8bcd1e) + 516
 bf8bcd56 _thrp_setup (bfdf0a00) + 7e
 bf8bcfe0 _lwp_start (bfdf0a00, 0, 0, bf8bcd1e, 0, 0)

+++++++++++++++++++++++++++++++++++++++

Other core dump for sp_notembedded.test

<suffrene:mysql-test>pstack /var/tmp/var/master-data/core
core '/var/tmp/var/master-data/core' of 2489:   /usr/mysql/5.0/bin/mysqld --no-defaults --basedir=/usr/mysql/5.0 --cha
-----------------  lwp# 1 / thread# 1  --------------------
 bf8c1687 __pollsys (8045a70, 2, 0, 0) + 7
 bf86ce01 pselect  (9, 8045c20, 0, 0, 0, 0) + 199
 bf86d1d6 select   (9, 8045c20, 0, 0, 0, 89254ac) + 78
 08242b1c handle_connections_sockets (0) + 324
 0824223f main     (1e, 80466b4, 8046730, 81852f0) + 111f
 0818538d _start   (1e, 804693c, 8046956, 8046964, 804697d, 80469b6) + 7d
-----------------  lwp# 2 / thread# 2  --------------------
 bf8c0fc7 __sigtimedwait (bfe3ef84, 0, 0, bf8ae9c7, 8921c88) + 7
 bf8ae9dd sigwait  (bfe3ef84, bf94f000, bfe3ef5c, bf89871b) + 22
 bf89872c __posix_sigwait (bfe3ef84, bfe3ef78, 0, 14) + 34
 0823f503 signal_hand (0, bf94f000, bfe3efec, bf8bcd1e) + 1c7
 bf8bcd56 _thrp_setup (bfdf0200) + 7e
 bf8bcfe0 _lwp_start (bfdf0200, 0, 0, bf8bcd1e, 0, 0)
-----------------  lwp# 3 / thread# 3  --------------------
 bf87e435 _ndoprnt (86ede01, bfb4fa4c, bfb4fa24, 0) + c
 bf882e91 sprintf  (98615a9, 86ede01, 4, 9861578, 1, 86eddfe) + 39
 083f57f5 __1cHsp_nameKinit_qname6MpnDTHD__v_ (9861580, 8b90028, bfb53498, 826bf4d) + 85
 08276d3d __1cKMYSQdDLparse6Fpv_i_ (8b90028, 8b90028, 8c8c230, a4) + ae01
 08400a55 __1cPdb_load_routine6FpnDTHD_ipnHsp_name_ppnHsp_head_Lpkc88rnOst_sp_chistics_8xx_i_ (8b90028, 2, 985fc30, bfb54780, 0, 8ca4b90) + 299
 084029d7 __1cPsp_find_routine6FpnDTHD_ipnHsp_name_ppnIsp_cache_b_pnHsp_head__ (8b90028, 2, 985fc30, 8b90c2c, 1, 0) + 1a7
 082606ff __1cVmysql_execute_command6FpnDTHD__i_ (8b90028, 0, 4, 83fa604) + 4127
 083fa616 __1cNsp_instr_stmtJexec_core6MpnDTHD_pI_i_ (985fe78, 8b90028, bfb55a70, 49) + 1e
 083fa251 __1cNsp_instr_stmtHexecute6MpnDTHD_pI_i_ (985fe78, 8b90028, bfb55a70, 83f6211) + 6cd
................
 083f65bf __1cHsp_headHexecute6MpnDTHD__b_ (8c04038, 8b90028, 8b90865, 8bfc1f4) + 3bf
 083f79da __1cHsp_headRexecute_procedure6MpnDTHD_pnEList4nEItem____b_ (8c04038, 8b90028, 8b91150, 0) + 3f6
 0826088f __1cVmysql_execute_command6FpnDTHD__i_ (8b90028, 8b90028, 8bfc028, 19) + 42b7
 0825aa8b __1cQdispatch_command6FnTenum_server_command_pnDTHD_pcI_b_ (3, 8b90028, 8bb4019, 1a) + 2b9b
 08256f96 handle_one_connection (8b90028, bf94f000, bfb7efec, bf8bcd1e) + 516
 bf8bcd56 _thrp_setup (bfdf0a00) + 7e
 bf8bcfe0 _lwp_start (bfdf0a00, 0, 0, bf8bcd1e, 0, 0)
[19 Mar 2009 8:14] Sveta Smirnova
There is similar bug #28271, but for Netware
[19 Mar 2009 8:25] Sveta Smirnova
Thank you for the report.

I can not repeat described behavior on Solaris 10. Additionally version 5.0.67 is old. Please try with current version 5.0.77 and if problem still exists provide accurate version of MySQL package you use (file name) and accurate version of Solaris.
[19 Mar 2009 8:33] Sunanda Menon
Please check the bug against OpenSolaris nv_b110 for x86
[19 Mar 2009 9:50] Sveta Smirnova
Thank you for the feedback.

I can not repeat described behavior on OpenSolaris using version 5.0.77 as well. Please check in your environment if bug does not exists in version 5.0.77 to be sure it is fixed.
[19 Mar 2009 9:57] Sunanda Menon
No ,this bug does exists in 5.0.77 also on opensolaris -x86 unless of-course I'm doing something differently which you may feel free to point out .
This is how I reproduce the bug and look at the core file that generates.
Please note this system has b107 of nevada ,I'm sure it should be the same in b110 as well.

Check details below
--------------------

cd /usr/mysql/5.0/mysql-test
$ ./mysql-test-run  --vardir=/var/tmp/var --verbose t/execution_constants.test
Logging: ./mysql-test-run --vardir=/var/tmp/var --verbose t/execution_constants.test
> exe_name: mysqld
MySQL Version 5.0.77
Using ndbcluster when necessary, mysqld supports it
Setting mysqld to support SSL connections
Using MTR_BUILD_THREAD      = 0
Using MASTER_MYPORT         = 9306
Using MASTER_MYPORT1        = 9307
Using SLAVE_MYPORT          = 9308
Using SLAVE_MYPORT1         = 9309
Using SLAVE_MYPORT2         = 9310
Using NDBCLUSTER_PORT       = 9311
Using IM_PORT               = 9313
Using IM_MYSQLD1_PORT       = 9314
Using IM_MYSQLD2_PORT       = 9315
Killing Possible Leftover Processes
> mtr_mysqladmin_start, pid: 11977
> mtr_mysqladmin_start, pid: 11978
> mtr_mysqladmin_start, pid: 11979
> mtr_mysqladmin_start, pid: 11980
> mtr_mysqladmin_start, pid: 11981
> mtr_wait_blocking
> mtr_ping_port: 9306
> FREE
> mtr_ping_port: 9307
> FREE
> mtr_ping_port: 9308
> FREE
> mtr_ping_port: 9309
> FREE
> mtr_ping_port: 9310
> FREE
Removing Stale Files
> opt_vardir: /var/tmp/var
> Removing /usr/mysql/5.0/mysql-test/var
> mtr_rmtree: /usr/mysql/5.0/mysql-test/var
> Removing /var/tmp/var/
> mtr_rmtree: /var/tmp/var/
Creating Directories
> Creating /var/tmp/var
Installing Master Database
> result: , file_mode: 0
=======================================================
> Starting timer for 'suite', duration: 10800, pid: 11983
Starting Tests in the 'main' suite

TEST                           RESULT         TIME (ms)
-------------------------------------------------------

> Starting timer for 'testcase', duration: 900, pid: 11984
> Setting timezone: GMT-3
> Restart master: master is not started
> Skip slave restart: No testcase use slaves
> mysqld pid: 11985
execution_constants            [ fail ]

mysqltest: At line 68: query '$query_head 0 $query_tail' failed with wrong errno 2013: 'Lost connection to MySQL server during query', instead of 0...

The result from queries just before the failure was:
CREATE TABLE `t_bug21476` (
`ID_BOARD` smallint(5) unsigned NOT NULL default '0',
`ID_MEMBER` mediumint(8) unsigned NOT NULL default '0',
`logTime` int(10) unsigned NOT NULL default '0',
`ID_MSG` mediumint(8) unsigned NOT NULL default '0',
PRIMARY KEY  (`ID_MEMBER`,`ID_BOARD`),
KEY `logTime` (`logTime`)
) ENGINE=MyISAM DEFAULT CHARSET=cp1251 COLLATE=cp1251_bulgarian_ci;
INSERT INTO `t_bug21476` VALUES (2,2,1154870939,0),(1,2,1154870957,0),(2,183,1154941362,0),(2,84,1154904301,0),(1,84,1154905867,0),(2,13,1154947484,10271),(3,84,1154880549,0),(1,6,1154892183,0),(2,25,1154947581,10271),(3,25,1154904760,0),(1,25,1154947373,10271),(1,179,1154899992,0),(2,179,1154899410,0),(5,25,1154901666,0),(2,329,1154902026,0),(3,329,1154902040,0),(1,329,1154902058,0),(1,13,1154930841,0),(3,85,1154904987,0),(1,183,1154929665,0),(3,13,1154931268,0),(1,85,1154936888,0),(1,169,1154937959,0),(2,169,1154941717,0),(3,183,1154939810,0),(3,169,1154941734,0);

More results from queries before failure can be found in /var/tmp/var/log/execution_constants.log

Aborting: execution_constants failed in default mode.
To continue, re-run with '--force'.
Stopping All Servers
> mtr_mysqladmin_start, pid: 11987
> mtr_wait_blocking
> mtr_check_stop_servers
> mtr_ping_port: 9306
> FREE
> Caught exit of process 11985
> Stopping timer for 'suite' with pid 11983
> timer 11983 woke up, exiting!
> Stopping timer for 'testcase' with pid 11984
> timer 11984 woke up, exiting!
$ ls -l /var/tmp/var
total 10
drwxr-xr-x   2 mysql    mysql        512 Mar 19 02:48 log
drwxr-xr-x   4 mysql    mysql        512 Mar 19 02:48 master-data
drwxr-xr-x   2 mysql    mysql        512 Mar 19 02:48 run
lrwxrwxrwx   1 mysql    mysql         34 Mar 19 02:48 std_data_ln -> /usr/mysql/5.0/mysql-test/std_data
drwxr-xr-x   2 mysql    mysql        512 Mar 19 02:48 tmp
$ ls -l /var/tmp/var/master-data
total 58822
-rw-------   1 mysql    mysql    30086991 Mar 19 02:48 core
drwxr-xr-x   2 mysql    mysql       1536 Mar 19 02:48 mysql
drwxr-xr-x   2 mysql    mysql        512 Mar 19 02:48 test

$ uname -a
SunOS htpasswd 5.11 snv_107 i86pc i386 i86pc
$ cat /etc/release
                  Solaris Express Community Edition snv_107 X86
           Copyright 2009 Sun Microsystems, Inc.  All Rights Reserved.
                        Use is subject to license terms.
                            Assembled 26 January 2009
$ /usr/mysql/5.0/bin/mysql
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 1
Server version: 5.0.77 Source distribution

Type 'help;' or '\h' for help. Type '\c' to clear the buffer.

mysql>
[19 Mar 2009 10:31] Sunanda Menon
The packages has been built using Sun Studio 12 
Configure options used are as follows

                --prefix=$(PREFIX) \
                --localstatedir=$(DATA_PREFIX)/data \
                --datadir=$(PREFIX)/share  \
                --sbindir=$(PREFIX)sbin  \
                --sharedstatedir=$(PREFIX)/com  \
                --includedir=$(PREFIX)/include \
                --oldincludedir=$(PREFIX)/include \
                --infodir=$(PREFIX)/docs \
                --mandir=$(PREFIX)/man  \
                --sysconfdir=$(CONFDIR)  \
                --enable-thread-safe-client     \
                --with-mysqld-libs=-lmtmalloc   \
                --with-named-curses=-lcurses    \
                --with-client-ldflags=-static   \
                --with-mysql-ldflags=-static    \
                --with-pic \
                --with-big-tables \
                --with-openssl=/ \
                --with-openssl-includes=/usr/include \
                --with-openssl-libs=/lib \
                --with-readline \
                --with-archive-storage-engine   \
                --with-blackhole-storage-engine \
                --with-csv-storage-engine       \
                --with-example-storage-engine   \
                --with-federated-storage-engine \
                --with-innodb   \
                --with-extra-charsets=complex   \
                --enable-local-infile  \
                --with-ndbcluster
[20 May 2009 20:20] Sveta Smirnova
Tried with OpenSolaris b111, still can not repeat.

So I close the report. Feel free to reopen the report if you meet it again.
[30 Jul 2009 10:25] Sunanda Menon
The problem is seen again in 5.0.81 on OpenSolaris b119 x86.
Please log into suffrene.france.sun.com with Sun/LDAP account and verify the test failure as 
cd /usr/mysql/mysql-test
/usr/mysql/mysql-test/mysql-test-run  --vardir=/tmp/var --verbose t/execution_constants.test

Tests that are dumping core are namely
execution_constants
sp_notembedded
subselect

Please note mysqld is running.
[4 Aug 2009 9:49] Sveta Smirnova
Thank you for the report.

Verified as described using your build.

Although when I tried to check with newer version, then compiled 5.0.81 on same box neither crashes.
[7 Aug 2009 9:08] Sveta Smirnova
Omer,

no, thread_stack doesn't matter.

Sunanda,

as problem is repeatable only with your build could you please try with current version 5.0.84 and inform us if problem still exists?
[7 Sep 2009 23:00] Bugs System
No feedback was provided for this bug for over a month, so it is
being suspended automatically. If you are able to provide the
information that was originally requested, please do so and change
the status of the bug back to "Open".
[10 Sep 2009 5:07] Shane Bester
you should try setting STACK_MIN_SIZE to a much larger value in mysql_priv.h and rebuild - it should then pass execution_constants.test..
[3 Dec 2009 7:55] Sunanda Menon
I have tried increasing the STACK_MIZ_SIZE=16000 and see that execution_constants test still fails.Will try increasing to 18000 as my last try,but is there any basis by which this needs to increase as this doesn't seem to be the right fix for me.I still see the failure on Opensolaris(X86)
Please note the mysql release versions in Opensolaris lags behind the mysql releases.Currently we have 5.0.86 in Opensolaris and there is a bug associated with that on bugster CR6818295.
[9 Dec 2009 7:14] Sunanda Menon
This looks more like -xO4 flag which is causing this test to fail and I have reproduced this.Using -xO2 flag works fine.
More related to http://bugs.mysql.com/bug.php?id=49091
[16 Dec 2009 16:46] Kent Boortz
This is not about stack size, but a failing stack direction test in the configure script.
The MySQL server uses knowledge about if the stack grows to lower or higher addresses
to be able to do tests to avoid stack overflow.

The added optimization going from -xO3 to -xO4 changes the result from running
a C code snippet in configure, a function gets inlined and then the comparison
of pointers into the stack doesn't indicate the direction any longer. And as it happens
in this case, -xO4 will make configure think the stack growth is in the opposite direction
from what it really is.

One solution to solve this is to extend on the change done for Bug#42213, and add for
Sun Studio to the test in "config/ac-macros/misc.m4" the following

  #ifdef __SUNPRO_C
  int find_stack_direction (void);
  #pragma no_inline (find_stack_direction)
  #endif