Bug #45912 "stack smashing" in ndbd

Submitted:	2 Jul 2009 13:02	Modified:	14 Sep 2009 13:06
Reporter:	Sven Sandberg	Email Updates:
Status:	Verified	Impact on me:	None
Category:	MySQL Cluster: Cluster (NDB) storage engine	Severity:	S1 (Critical)
Version:	mysql-5.1	OS:	Linux (Ubuntu Jaunty)
Assigned to:		CPU Architecture:	Any
Tags:	5.1, ndb, stack smashing detected

Description:
When running simple test cases using ndb, I sometimes get this error message:

*** stack smashing detected ***: /home/sven/bzr/debug-max/5.1-bugteam/storage/ndb/src/kernel/ndbd terminated
======= Backtrace: =========
/lib/tls/i686/cmov/libc.so.6(__fortify_fail+0x48)[0xb7e60da8]
/lib/tls/i686/cmov/libc.so.6(__fortify_fail+0x0)[0xb7e60d60]
/home/sven/bzr/debug-max/5.1-bugteam/storage/ndb/src/kernel/ndbd[0x83250b5]
/home/sven/bzr/debug-max/5.1-bugteam/storage/ndb/src/kernel/ndbd(_ZN13ErrorReporter11handleErrorEiPKcS1_15NdbShutdownType+0x30)[0x83250e8]
/home/sven/bzr/debug-max/5.1-bugteam/storage/ndb/src/kernel/ndbd(_ZNK14SimulatedBlock9progErrorEiiPKc+0x11c)[0x830eea6]
/home/sven/bzr/debug-max/5.1-bugteam/storage/ndb/src/kernel/ndbd(_ZN4Qmgr15stateArbitCrashEP6Signal+0x11f)[0x829b5fb]
/home/sven/bzr/debug-max/5.1-bugteam/storage/ndb/src/kernel/ndbd(_ZN4Qmgr14runArbitThreadEP6Signal+0x22f)[0x829fd5d]
/home/sven/bzr/debug-max/5.1-bugteam/storage/ndb/src/kernel/ndbd(_ZN4Qmgr16startArbitThreadEP6Signal+0xd7)[0x829ffb1]
/home/sven/bzr/debug-max/5.1-bugteam/storage/ndb/src/kernel/ndbd(_ZN4Qmgr16handleArbitCheckEP6Signal+0x56a)[0x82a11f4]
/home/sven/bzr/debug-max/5.1-bugteam/storage/ndb/src/kernel/ndbd(_ZN4Qmgr17execPREP_FAILCONFEP6Signal+0x209)[0x82a1407]
/home/sven/bzr/debug-max/5.1-bugteam/storage/ndb/src/kernel/ndbd(_ZN14SimulatedBlock15executeFunctionEtP6Signal+0xd3)[0x81202f9]
/home/sven/bzr/debug-max/5.1-bugteam/storage/ndb/src/kernel/ndbd(_ZN13FastScheduler5doJobEv+0x1bd)[0x8316001]
/home/sven/bzr/debug-max/5.1-bugteam/storage/ndb/src/kernel/ndbd(_ZN12ThreadConfig13ipControlLoopEv+0x10b)[0x83170c7]
/home/sven/bzr/debug-max/5.1-bugteam/storage/ndb/src/kernel/ndbd(main+0x941)[0x80fa1a5]
/lib/tls/i686/cmov/libc.so.6(__libc_start_main+0xe5)[0xb7d79775]
/home/sven/bzr/debug-max/5.1-bugteam/storage/ndb/src/kernel/ndbd[0x80f8a31]
======= Memory map: ========
08048000-083d2000 r-xp 00000000 08:04 4849860    /home/sven/bzr/debug-max/5.1-bugteam/storage/ndb/src/kernel/ndbd
083d2000-083d9000 r--p 00389000 08:04 4849860    /home/sven/bzr/debug-max/5.1-bugteam/storage/ndb/src/kernel/ndbd
083d9000-084dd000 rw-p 00390000 08:04 4849860    /home/sven/bzr/debug-max/5.1-bugteam/storage/ndb/src/kernel/ndbd
084dd000-0851d000 rw-p 084dd000 00:00 0 
08a88000-08dde000 rw-p 08a88000 00:00 0          [heap]
b0b84000-b127e000 rw-p b0b84000 00:00 0 
b127e000-b127f000 ---p b127e000 00:00 0 
b127f000-b12c3000 rw-p b127f000 00:00 0 
b12c3000-b12c4000 ---p b12c3000 00:00 0 
b12c4000-b1308000 rw-p b12c4000 00:00 0 
b1308000-b1309000 ---p b1308000 00:00 0 
b1309000-b134d000 rw-p b1309000 00:00 0 
b134d000-b134e000 ---p b134d000 00:00 0 
b134e000-b1392000 rw-p b134e000 00:00 0 
b1392000-b1393000 ---p b1392000 00:00 0 
b1393000-b13d7000 rw-p b1393000 00:00 0 
b13d7000-b13d8000 ---p b13d7000 00:00 0 
b13d8000-b141c000 rw-p b13d8000 00:00 0 
b141c000-b141d000 ---p b141c000 00:00 0 
b141d000-b1461000 rw-p b141d000 00:00 0 
b1461000-b1462000 ---p b1461000 00:00 0 
b1462000-b14a6000 rw-p b1462000 00:00 0 
b14a6000-b14a7000 ---p b14a6000 00:00 0 
b14a7000-b14eb000 rw-p b14a7000 00:00 0 
b14eb000-b14ec000 ---p b14eb000 00:00 0 
b14ec000-b1530000 rw-p b14ec000 00:00 0 
b1530000-b1531000 ---p b1530000 00:00 0 
b1531000-b1575000 rw-p b1531000 00:00 0 
b1575000-b1576000 ---p b1575000 00:00 0 
b1576000-b15ba000 rw-p b1576000 00:00 0 
b15ba000-b15bb000 ---p b15ba000 00:00 0 
b15bb000-b15ff000 rw-p b15bb000 00:00 0 
b15ff000-b1600000 ---p b15ff000 00:00 0 
b1600000-b1644000 rw-p b1600000 00:00 0 
b1644000-b1645000 ---p b1644000 00:00 0 
b1645000-b1689000 rw-p b1645000 00:00 0 
b1689000-b168a000 ---p b1689000 00:00 0 
b168a000-b16ce000 rw-p b168a000 00:00 0 
b16ce000-b16cf000 ---p b16ce000 00:00 0 
b16cf000-b1713000 rw-p b16cf000 00:00 0 
b1713000-b1714000 ---p b1713000 00:00 0 
b1714000-b1758000 rw-p b1714000 00:00 0 
b1758000-b1759000 ---p b1758000 00:00 0 
b1759000-b179d000 rw-p b1759000 00:00 0 
b179d000-b179e000 ---p b179d000 00:00 0 
b179e000-b17e2000 rw-p b179e000 00:00 0 
b17e2000-b17e3000 ---p b17e2000 00:00 0 
b17e3000-b1827000 rw-p b17e3000 00:00 0 
b1827000-b1828000 ---p b1827000 00:00 0 
b1828000-b186c000 rw-p b1828000 00:00 0 
b186c000-b186d000 ---p b186c000 00:00 0 
b186d000-b18b1000 rw-p b186d000 00:00 0 
b18b1000-b18b2000 ---p b18b1000 00:00 0 
b18b2000-b18f6000 rw-p b18b2000 00:00 0 
b18f6000-b18f7000 ---p b18f6000 00:00 0 
b18f7000-b193b000 rw-p b18f7000 00:00 0 
b193b000-b193c000 ---p b193b000 00:00 0 
b193c000-b1980000 rw-p b193c000 00:00 0 
b1980000-b1981000 ---p b1980000 00:00 0 
b1981000-b7172000 rw-p b1981000 00:00 0 
b7172000-b7173000 ---p b7172000 00:00 0 
b7173000-b717a000 rw-p b7173000 00:00 0 
b717a000-b717b000 ---p b717a000 00:00 0 
b717b000-b7182000 rw-p b717b000 00:00 0 
b7182000-b7183000 ---p b7182000 00:00 0 
b7183000-b741b000 rw-p b7183000 00:00 0 
b741b000-b7425000 r-xp 00000000 08:02 521254     /lib/tls/i686/cmov/libnss_files-2.9.so
b7425000-b7426000 r--p 00009000 08:02 521254     /lib/tls/i686/cmov/libnss_files-2.9.so
b7426000-b7427000 rw-p 0000a000 08:02 521254     /lib/tls/i686/cmov/libnss_files-2.9.so
b7427000-b7434000 r-xp 00000000 08:02 488685     /lib/libgcc_s.so.1
b7434000-b7435000 r--p 0000c000 08:02 488685     /lib/libgcc_s.so.1
b7435000-b7436000 rw-p 0000d000 08:02 488685     /lib/libgcc_s.so.1
b7436000-b7437000 ---p b7436000 00:00 0 
b7437000-b7d63000 rw-p b7437000 00:00 0 
b7d63000-b7ebf000 r-xp 00000000 08:02 521240     /lib/tls/i686/cmov/libc-2.9.so
b7ebf000-b7ec0000 ---p 0015c000 08:02 521240     /lib/tls/i686/cmov/libc-2.9.so
b7ec0000-b7ec2000 r--p 0015c000 08:02 521240     /lib/tls/i686/cmov/libc-2.9.so
b7ec2000-b7ec3000 rw-p 0015e000 08:02 521240     /lib/tls/i686/cmov/libc-2.9.so
b7ec3000-b7ec6000 rw-p b7ec3000 00:00 0 
b7ec6000-b7eea000 r-xp 00000000 08:02 521249     /lib/tls/i686/cmov/libm-2.9.so
b7eea000-b7eeb000 r--p 00023000 08:02 521249     /lib/tls/i686/cmov/libm-2.9.so
b7eeb000-b7eec000 rw-p 00024000 08:02 521249     /lib/tls/i686/cmov/libm-2.9.so
b7eec000-b7f01000 r-xp 00000000 08:02 521251     /lib/tls/i686/cmov/libnsl-2.9.so
b7f01000-b7f02000 r--p 00014000 08:02 521251     /lib/tls/i686/cmov/libnsl-2.9.so
b7f02000-b7f03000 rw-p 00015000 08:02 521251     /lib/tls/i686/cmov/libnsl-2.9.so
b7f03000-b7f05000 rw-p b7f03000 00:00 0 
b7f05000-b7f0e000 r-xp 00000000 08:02 521247     /lib/tls/i686/cmov/libcrypt-2.9.so
b7f0e000-b7f0f000 r--p 00008000 08:02 521247     /lib/tls/i686/cmov/libcrypt-2.9.so
b7f0f000-b7f10000 rw-p 00009000 08:02 521247     /lib/tls/i686/cmov/libcrypt-2.9.so
b7f10000-b7f38000 rw-p b7f10000 00:00 0 
b7f38000-b7f4d000 r-xp 00000000 08:02 521259     /lib/tls/i686/cmov/libpthread-2.9.so
b7f4d000-b7f4e000 r--p 00014000 08:02 521259     /lib/tls/i686/cmov/libpthread-2.9.so
b7f4e000-b7f4f000 rw-p 00015000 08:02 521259     /lib/tls/i686/cmov/libpthread-2.9.so
b7f4f000-b7f51000 rw-p b7f4f000 00:00 0 
b7f62000-b7f67000 rw-p b7f62000 00:00 0 
b7f67000-b7f68000 r-xp b7f67000 00:00 0          [vdso]
b7f68000-b7f84000 r-xp 00000000 08:02 488671     /lib/ld-2.9.so
b7f84000-b7f85000 r--p 0001b000 08:02 488671     /lib/ld-2.9.so
b7f85000-b7f86000 rw-p 0001c000 08:02 488671     /lib/ld-2.9.so
bf96f000-bf986000 rw-p bffe9000 00:00 0          [stack]
090702 14:48:23 [Note] Plugin 'FEDERATED' is disabled.
090702 14:48:23 [Note] Plugin 'ndbcluster' is disabled.
090702 14:48:23 [Warning] Forcing shutdown of 2 plugins

How to repeat:
I can reproduce this maybe 2-10% of the times, using the following test case:

source include/have_ndb.inc;
create table t1 (a int) engine = ndb;
insert into t1 values (1);
drop table t1;
exit;

There is nothing special with this test case, I have seen it when running other tests too.

I execute the test case as follows:

cd mysql-test
n=0
while [ 1 ] ; do
  n=$((n+1))
  echo ==== $n ====
  ./mtr ndb_stack_smashing
done

So far it has failed 3 times out of 38.

Thank you for the report.

I can not repeat described behavior with current 5.1-bugteam tree. Please try it now and if you still able to repeat answer Jørgen's questions

I can still repeat this. I pulled today's 5.1-bugteam, revid:luis.soares@sun.com-20090913214347-nvzyipdt40e21rx7 and ran the same test with similar output.

I have built using ./BUILD/compile-pentium-debug-max .

I'm using Ubuntu Jaunty.

$ uname -a
Linux riska 2.6.28-15-generic #49-Ubuntu SMP Tue Aug 18 18:40:08 UTC 2009 i686 GNU/Linux

Let me know if you need anything else!

Page generated in 0.040 sec. using MySQL 8.4.7-u6-cloud

Timestamp references displayed by the system are UTC.
Content reproduced on this site is the property of the respective copyright holders.
It is not reviewed in advance by Oracle and does not necessarily represent the opinion of Oracle or any other party.