Bug #46009 mysqld crashes caused by ndb$info query
Submitted: 7 Jul 2009 13:28 Modified: 10 Sep 2009 12:21
Reporter: Wen Xiong Email Updates:
Status: No Feedback Impact on me:
None 
Category:MySQL Cluster: Cluster (NDB) storage engine Severity:S2 (Serious)
Version:mysql-5.1-telco-7.1 OS:Linux
Assigned to: Magnus BlÄudd CPU Architecture:Any
Triage: Triaged: D2 (Serious)

[7 Jul 2009 13:28] Wen Xiong
Description:
mysql> select * from POOLS order by POOL_NAME;
ERROR 2013 (HY000): Lost connection to MySQL server during query

Then mysqld crashes and can not be restarted.

After I restart mysqld like before:
libexec/mysqld --defaults-group-suffix=.1.1 --defaults-file=/home/wx228566/mysql2/mysql-5.1-telco-7.0_NDBINFO/linux2/mysql-test/var/my.cnf --log-output=table,file --loose-skip-innodb --binlog-format=row --core-file --plugin_dir=/home/wx228566/mysql2/mysql-5.1-telco-7.0_NDBINFO/linux/lib/mysql/plugin

it can not be restarted and error message is shown below:

090629 14:58:00 [Warning] The syntax '--log' is deprecated and will be removed in MySQL 7.0. Please use '--general_log'/'--general_log_file' instead.
090629 14:58:00 [Warning] The syntax '--log_slow_queries' is deprecated and will be removed in MySQL 7.0. Please use '--slow_query_log'/'--slow_query_log_file' instead.
090629 14:58:00 [Note] Plugin 'InnoDB' disabled by command line option
090629 14:58:00 [Note] NDB: NodeID is 4, management server 'loki43:12500'
090629 14:58:01 [Note] NDB[0]: NodeID: 4, all storage nodes connected
090629 14:58:01 [Note] NDB[1]: NodeID: 5, all storage nodes connected
090629 14:58:01 [Note] NDB[2]: NodeID: 6, all storage nodes connected
090629 14:58:01 [Note] Starting Cluster Binlog Thread
090629 14:58:01 - mysqld got signal 11 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help diagnose
the problem, but since we have already crashed, something is definitely wrong
and this may fail.

key_buffer_size=1048576
read_buffer_size=131072
max_used_connections=0
max_threads=151
threads_connected=0
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 60551 K
bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

thd: 0x0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = (nil) thread_stack 0x40000
/home/wx228566/mysql2/mysql-5.1-telco-7.0_NDBINFO/linux2/libexec/mysqld(my_print_stacktrace+0x39)[0x98e0d9]
/home/wx228566/mysql2/mysql-5.1-telco-7.0_NDBINFO/linux2/libexec/mysqld(handle_segfault+0x31d)[0x6341cd]
/lib64/libpthread.so.0[0x3275a0de70]
/home/wx228566/mysql2/mysql-5.1-telco-7.0_NDBINFO/linux/lib/mysql/plugin/ha_ndbcluster_info.so[0x2aaaaac08cc3]
/home/wx228566/mysql2/mysql-5.1-telco-7.0_NDBINFO/linux2/libexec/mysqld(_Z24ha_initialize_handlertonP13st_plugin_int+0x38)[0x72b408]
/home/wx228566/mysql2/mysql-5.1-telco-7.0_NDBINFO/linux2/libexec/mysqld[0x7af092]
/home/wx228566/mysql2/mysql-5.1-telco-7.0_NDBINFO/linux2/libexec/mysqld(_Z11plugin_initPiPPci+0x87c)[0x7b25ac]
/home/wx228566/mysql2/mysql-5.1-telco-7.0_NDBINFO/linux2/libexec/mysqld[0x634e32]
/home/wx228566/mysql2/mysql-5.1-telco-7.0_NDBINFO/linux2/libexec/mysqld(main+0x699)[0x638b89]
/lib64/libc.so.6(__libc_start_main+0xf4)[0x3274e1d8b4]
/home/wx228566/mysql2/mysql-5.1-telco-7.0_NDBINFO/linux2/libexec/mysqld[0x564e29]
The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you find out what is causing the crash.
Writing a core file
Segmentation fault (core dumped)

So I debug the core file, and it says:
gdb ./libexec/mysqld --core=mysql-test/var/mysql_cluster.1/mysqld.1/data/core.21269

GNU gdb Red Hat Linux (6.5-37.el5rh)
Copyright (C) 2006 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu"...Using host libthread_db library "/lib64/libthread_db.so.1".

Reading symbols from /lib64/libpthread.so.0...done.
Loaded symbols for /lib64/libpthread.so.0
Reading symbols from /lib64/libdl.so.2...done.
Loaded symbols for /lib64/libdl.so.2
Reading symbols from /lib64/librt.so.1...done.
Loaded symbols for /lib64/librt.so.1
Reading symbols from /lib64/libcrypt.so.1...done.
Loaded symbols for /lib64/libcrypt.so.1
Reading symbols from /lib64/libnsl.so.1...done.
Loaded symbols for /lib64/libnsl.so.1
Reading symbols from /lib64/libm.so.6...done.
Loaded symbols for /lib64/libm.so.6
Reading symbols from /lib64/libc.so.6...done.
Loaded symbols for /lib64/libc.so.6
Reading symbols from /lib64/ld-linux-x86-64.so.2...done.
Loaded symbols for /lib64/ld-linux-x86-64.so.2
Reading symbols from /lib64/libgcc_s.so.1...done.
Loaded symbols for /lib64/libgcc_s.so.1
Reading symbols from /home/wx228566/mysql2/mysql-5.1-telco-7.0_NDBINFO/linux/lib/mysql/plugin/ha_ndbcluster_info.so...done.
Loaded symbols for /home/wx228566/mysql2/mysql-5.1-telco-7.0_NDBINFO/linux/lib/mysql/plugin/ha_ndbcluster_info.so
Core was generated by `/home/wx228566/mysql2/mysql-5.1-telco-7.0_NDBINFO/linux2/libexec/mysqld --defau'.
Program terminated with signal 11, Segmentation fault.
#0  0x0000003275a0b122 in pthread_kill () from /lib64/libpthread.so.0
(gdb) where
#0  0x0000003275a0b122 in pthread_kill () from /lib64/libpthread.so.0
#1  0x0000000000634201 in handle_segfault (sig=11) at mysqld.cc:2576
#2  <signal handler called>
#3  ndbcluster_info_init_func (p=0x3e15310) at ha_ndbcluster_info.cc:293
#4  0x000000000072b408 in ha_initialize_handlerton (plugin=0x3d747c8)
    at handler.cc:435
#5  0x00000000007af092 in plugin_initialize (plugin=0x2aaaaac0abcf)
    at sql_plugin.cc:1006
#6  0x00000000007b25ac in plugin_init (argc=0xf5fdd0, argv=0x3d54e50,
    flags=<value optimized out>) at sql_plugin.cc:1219
#7  0x0000000000634e32 in init_server_components () at mysqld.cc:3922
#8  0x0000000000638b89 in main (argc=<value optimized out>,
    argv=<value optimized out>) at mysqld.cc:4394

How to repeat:
start cluster
create database ndb$info;

CREATE TABLE `TABLES` (
  `TABLE_ID` INT UNSIGNED,
  `TABLE_NAME` VARCHAR(512),
  `CREATE_SQL` VARCHAR(512)
  ) ENGINE=NDBINFO;

CREATE TABLE `POOLS` (
  `NODE_ID` INT UNSIGNED,
  `BLOCK_NUMBER` INT UNSIGNED,
  `BLOCK_INSTANCE` INT UNSIGNED,
  `POOL_NAME` VARCHAR(512),
  `FREE` INT UNSIGNED,
  `SIZE` INT UNSIGNED
  ) ENGINE=NDBINFO;

select * from POOLS order by POOL_NAME;
[17 Jul 2009 8:41] Sveta Smirnova
Thank you for the report.

With today sources built with applied patch for bug #46113 I get error "query 'select * from POOLS order by POOL_NAME' failed: 1146: Table 'test.POOLS' doesn't exist" instead of crash.
[10 Aug 2009 12:21] Jonas Oreland
sveta tests is expected behaviour...
wen: retest on latest 7.1
[13 Aug 2009 20:01] Sveta Smirnova
Re-tested with today bzr tree, same expected results.

Wen, please try in your environment if no crash anymore.
[10 Sep 2009 23:00] Bugs System
No feedback was provided for this bug for over a month, so it is
being suspended automatically. If you are able to provide the
information that was originally requested, please do so and change
the status of the bug back to "Open".