MySQL Bugs: #40464: MySQL Cluster SQL Node Crashes during sysbench tests

Bug #40464	MySQL Cluster SQL Node Crashes during sysbench tests
Submitted:	31 Oct 2008 19:51	Modified:	24 Feb 2009 12:02
Reporter:	Andrew Rankin	Email Updates:
Status:	Closed	Impact on me:	None
Category:	MySQL Cluster: Cluster (NDB) storage engine	Severity:	S3 (Non-critical)
Version:	5.1.27-ndb-6.3.17,	OS:	Linux (2.6.18-92.1.13.el5 x86_64, RHEL 5.2)
Assigned to:	Martin Skold	CPU Architecture:	Any
Tags:	5.1.27-ndb-6.3.17-cluster-gpl, cluster, crash, sysbench

Description:
I've been testing out MySQL Cluster and have run into a scenario that always leads to a crash of mysqld on either of the SQL Nodes.  This is in the error log from the crash:

/usr/sbin/mysqld(print_stacktrace+0x1e)[0x72556e]
/usr/sbin/mysqld(handle_segfault+0x326)[0x60acd6]
/lib64/libpthread.so.0[0x2b13c3542e70]
/usr/sbin/mysqld[0x766c21]
/usr/sbin/mysqld(_ZN11Query_cache21send_result_to_clientEP3THDPcj+0x522)[0x711e42]
/usr/sbin/mysqld(_ZN18Prepared_statement7executeEP6Stringb+0x1a1)[0x687491]
/usr/sbin/mysqld(_ZN18Prepared_statement12execute_loopEP6StringbPhS2_+0x94)[0x688024]
/usr/sbin/mysqld(_Z18mysql_stmt_executeP3THDPcj+0xd5)[0x6882e5]
/usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcj+0x9e8)[0x61af18]
/usr/sbin/mysqld(_Z10do_commandP3THD+0xc7)[0x61ba07]
/usr/sbin/mysqld(handle_one_connection+0x6e0)[0x60f250]
/lib64/libpthread.so.0[0x2b13c353b2f7]
/lib64/libc.so.6(clone+0x6d)[0x2b13c4302b6d]
081031 13:23:49 - mysqld got signal 11 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help diagnose
the problem, but since we have already crashed, something is definitely wrong
and this may fail.

key_buffer_size=1048576000
read_buffer_size=2097152
max_used_connections=35
max_threads=2200
threads_connected=32
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 10044170 K
bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

thd: 0x27b74950
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
Trying to get some variables.
Some pointers may be invalid and cause the dump to abort...
thd->query at 0x2aac74008b00  is invalid pointer
thd->thread_id=305
thd->killed=NOT_KILLED
The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you find out what is causing the crash.

How to repeat:
I was testing with sysbench during the time it crashed.  We are using 4x Data Nodes, 2x SQL Nodes 1x Management Node.  

Steps to repeat are:

1) Load the database with sysbench data:

 sysbench --mysql-user=sbtest --mysql-db=sbtest --test=oltp --mysql-table-engine=ndbcluster --oltp-table-size=10000000 --mysql-socket=/var/lib/mysql/mysql.sock prepare

2) Run a read only sysbench run:

 sysbench --mysql-db=sbtest --num-threads=32 --max-requests=100000 --test=oltp --oltp-table-size=10000000 --mysql-socket=/var/lib/mysql/mysql.sock  --batch --batch-delay=5 --oltp-read-only run

3) Once that run finishes, re-run the same test.  The mysqld crash will occur immediately when sysbench kicks off.

Hi Andrew,

can you please attach your cluster and mysqld config files (config.ini and my.cnf). Maybe you can even upload the core file.

Verified with 6.3.18.

A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/65739

2813 Martin Skold	2009-02-10
      Bug#40464  MySQL Cluster SQL Node Crashes during sysbench tests

A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/65775

2836 Martin Skold	2009-02-10 [merge]
      Merge

A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/65920

3252 Martin Skold	2009-02-11 [merge]
      Merge

A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/65957

3257 Martin Skold	2009-02-11 [merge]
      Merge

Pushed into 5.1.31-ndb-6.4.3 (revid:martin.skold@mysql.com-20090211204523-03nx13fjekybwez2) (version source revid:martin.skold@mysql.com-20090211204523-03nx13fjekybwez2) (merge vers: 5.1.31-ndb-6.4.3) (pib:6)

Pushed into 5.1.31-ndb-6.2.17 (revid:martin.skold@mysql.com-20090210162313-g4ou3ovzoyoljaej) (version source revid:martin.skold@mysql.com-20090210124754-qu1i49ooicoi0skv) (merge vers: 5.1.31-ndb-6.2.17) (pib:6)

Pushed into 5.1.31-ndb-6.3.23 (revid:martin.skold@mysql.com-20090210163054-bcupy4stktagp5rt) (version source revid:martin.skold@mysql.com-20090210163054-bcupy4stktagp5rt) (merge vers: 5.1.31-ndb-6.3.23) (pib:6)

Documented in the NDB-6.2.17, 6.3.23, and 6.4.3 changelogs as follows:

        In some cases, NDB did not check correctly whether NDBCLUSTER
        tables had changed before trying to use the query cache. This
        could result in a crash of the debug MySQL server.