Description:
/home2/mydev/mysql-6.0-axmrg/mysql-test/mysql-test-run.pl --force --testcase-timeout=120 --suite-timeout=720 --ps-protocol --mysqld=--binlog-format=row
...
ndb.strict_autoinc_5ndb [ pass ] 1519
-------------------------------------------------------
Stopping All Servers
Warning; Aborted waiting on pid file: '/home2/mydev/testdir-6.0-axmrg-2/mysql-test/var/run/master.pid' after 70 seconds
All 672 tests were successful.
The servers were restarted 206 times
Spent 4042.443 of 5961 seconds executing testcases
mysql-test-run: WARNING: Got errors/warnings while running tests, please examine "/home2/mydev/testdir-6.0-axmrg-2/mysql-test/var/log/warnings" for details.
mysql-test-run: *** ERROR: there were failing test cases
warnings
========
master.err: ndb.strict_autoinc_5ndb: Attempting backtrace. You can use the following information to find out
master.err
==========
...
CURRENT_TEST: ndb.strict_autoinc_5ndb
080129 16:40:08 [Note] NDB Binlog: CREATE TABLE Event: REPL$test/t1
NDB: Found 5 NdbTransaction's that have not been released
NDB: Found 3 NdbReceiver's that have not been released
080129 16:40:09 [Note] Got signal 15 to shutdown mysqld
080129 16:40:09 [Note] /home2/mydev/testdir-6.0-axmrg-2/sql/mysqld: Normal shutdown
080129 16:40:09 [Note] Event Scheduler: Purging the queue. 0 events
080129 16:40:09 [Note] Stopping Cluster Binlog
NDB: Found 2 NdbTransaction's that have not been released
080129 16:40:09 [Note] Stopping Cluster Utility thread
NDB: table share ./mysql/ndb_apply_status with use_count 1 not freed
NDB: table share ./mysql/ndb_schema with use_count 1 not freed
080129 16:40:11 - mysqld got signal 6;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help diagnose
the problem, but since we have already crashed, something is definitely wrong
and this may fail.
key_buffer_size=1048576
read_buffer_size=131072
max_used_connections=2
max_threads=151
threads_connected=0
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 60523 K
bytes of memory
Hope that's ok; if not, decrease some variables in the equation.
thd: 0x0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
Cannot determine thread, fp=0x44028910, backtrace may not be correct.
Stack range sanity check OK, backtrace follows:
0x7091b2
0x2aaeecd8e025
0x1f823e0
New value of fp=0x44029950 failed sanity check, terminating stack trace!
Please read http://dev.mysql.com/doc/refman/5.1/en/resolve-stack-dump.html
and follow instructions on how to resolve the stack trace.
Resolved stack trace is much more helpful in diagnosing the
problem, so please do resolve it
The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you find out what is causing the crash.
Writing a core file
master1.err
===========
...
CURRENT_TEST: ndb.strict_autoinc_5ndb
NDB: Found 4 NdbTransaction's that have not been released
NDB: Found 2 NdbReceiver's that have not been released
080129 16:40:09 [Note] Got signal 15 to shutdown mysqld
080129 16:40:09 [Note] /home2/mydev/testdir-6.0-axmrg-2/sql/mysqld: Normal shutdown
080129 16:40:09 [Note] Event Scheduler: Purging the queue. 0 events
080129 16:40:09 [Note] Stopping Cluster Binlog
NDB: Found 2 NdbTransaction's that have not been released
080129 16:40:09 [Note] Stopping Cluster Utility thread
NDB: table share ./test/t1 with use_count 1 not freed
NDB: table share ./mysql/ndb_schema with use_count 1 not freed
NDB: table share ./mysql/ndb_apply_status with use_count 1 not freed
080129 16:40:11 [Note] /home2/mydev/testdir-6.0-axmrg-2/sql/mysqld: Shutdown complete
stack backtrace
===============
#4 0x00002aaeecd8e025 in raise () from /lib/libc.so.6
#5 0x00002aaeecd8fa80 in abort () from /lib/libc.so.6
#6 0x0000000000c46b20 in SendBuffer::bytesSent (this=0x1f2e230, bytes=36) at SendBuffer.hpp:106
#7 0x0000000000c45cb4 in TCP_Transporter::doSend (this=0x1f2dfd0) at TCP_Transporter.cpp:358
#8 0x0000000000c04ca1 in TransporterRegistry::performSend (this=0x1ed4410) at TransporterRegistry.cpp:889
#9 0x0000000000bf391f in TransporterFacade::threadMainSend (this=0x1eb2670) at TransporterFacade.cpp:484
#10 0x0000000000bf39a7 in runSendRequest_C (me=0x1eb2670) at TransporterFacade.cpp:466
#11 0x0000000000c2a370 in ndb_thread_wrapper (_ss=0x1ed6070) at NdbThread.c:81
#12 0x00002aaeeb92d3f7 in start_thread () from /lib/libpthread.so.0
#13 0x00002aaeece3397d in clone () from /lib/libc.so.6
100 SendBuffer::bytesSent(Uint32 bytes) {
101
102 if(bytes > dataSize){
103 #ifdef DEBUG_TRANSPORTER
104 printf("bytes(%d) > dataSize(%d)\n", bytes, dataSize);
105 #endif
106 abort();
bytes = 36
dataSize = 0
341 TCP_Transporter::doSend() {
...
352 const char * const sendPtr = m_sendBuffer.sendPtr;
353 const Uint32 sizeToSend = m_sendBuffer.sendDataSize;
354 if (sizeToSend > 0){
355 const int nBytesSent = send(theSocket, sendPtr, sizeToSend, 0);
356
357 if (nBytesSent > 0) {
358 m_sendBuffer.bytesSent(nBytesSent);
sizeToSend = 36
m_sendBuffer = {sizeOfBuffer = 262144, dataSize = 0, startOfBuffer = 0x1f42360, endOfBuffer = 0x1f82360, insertPtr = 0x1f42360, sendPtr = 0x1f42360 "$\023", sendDataSize = 0, theRemoteNodeId = 2}
I double checked. sizeToSend = 36, m_sendBuffer.sendDataSize = 0!
How to repeat:
It must be a random problem. I cannot repeat it. It happened once when I did the following:
OS: Debian GNU/Linux/x86_64
OS: Debian Sid kernel 2.6.23 SMP PREEMPT
gcc (GCC) 4.2.3 20080114 (prerelease) (Debian 4.2.2-7)
bk clone bk-internal.mysql.com:/home/bk/mysql-6.0-engines mysql-6.0-axmrg
cd mysql-6.0-axmrg
BUILD/compile-pentium64-debug-max --with-debug=full --prefix="/home2/mydev/install-6.0-axmrg"
cd mysql-test
./mysql-test-run.pl --force --testcase-timeout=120 --suite-timeout=720 --mysqld=--binlog-format=mixed
Suggested fix:
Is it possible that m_sendBuffer.sendDataSize is accessible and changable by more than one thread?
