Bug #45912 "stack smashing" in ndbd
Submitted: 2 Jul 2009 13:02 Modified: 14 Sep 2009 13:06
Reporter: Sven Sandberg Email Updates:
Status: Verified Impact on me:
None 
Category:MySQL Cluster: Cluster (NDB) storage engine Severity:S1 (Critical)
Version:mysql-5.1 OS:Linux (Ubuntu Jaunty)
Assigned to: CPU Architecture:Any
Tags: 5.1, ndb, stack smashing detected
Triage: Triaged: D2 (Serious) / R6 (Needs Assessment) / E6 (Needs Assessment)

[2 Jul 2009 13:02] Sven Sandberg
Description:
When running simple test cases using ndb, I sometimes get this error message:

*** stack smashing detected ***: /home/sven/bzr/debug-max/5.1-bugteam/storage/ndb/src/kernel/ndbd terminated
======= Backtrace: =========
/lib/tls/i686/cmov/libc.so.6(__fortify_fail+0x48)[0xb7e60da8]
/lib/tls/i686/cmov/libc.so.6(__fortify_fail+0x0)[0xb7e60d60]
/home/sven/bzr/debug-max/5.1-bugteam/storage/ndb/src/kernel/ndbd[0x83250b5]
/home/sven/bzr/debug-max/5.1-bugteam/storage/ndb/src/kernel/ndbd(_ZN13ErrorReporter11handleErrorEiPKcS1_15NdbShutdownType+0x30)[0x83250e8]
/home/sven/bzr/debug-max/5.1-bugteam/storage/ndb/src/kernel/ndbd(_ZNK14SimulatedBlock9progErrorEiiPKc+0x11c)[0x830eea6]
/home/sven/bzr/debug-max/5.1-bugteam/storage/ndb/src/kernel/ndbd(_ZN4Qmgr15stateArbitCrashEP6Signal+0x11f)[0x829b5fb]
/home/sven/bzr/debug-max/5.1-bugteam/storage/ndb/src/kernel/ndbd(_ZN4Qmgr14runArbitThreadEP6Signal+0x22f)[0x829fd5d]
/home/sven/bzr/debug-max/5.1-bugteam/storage/ndb/src/kernel/ndbd(_ZN4Qmgr16startArbitThreadEP6Signal+0xd7)[0x829ffb1]
/home/sven/bzr/debug-max/5.1-bugteam/storage/ndb/src/kernel/ndbd(_ZN4Qmgr16handleArbitCheckEP6Signal+0x56a)[0x82a11f4]
/home/sven/bzr/debug-max/5.1-bugteam/storage/ndb/src/kernel/ndbd(_ZN4Qmgr17execPREP_FAILCONFEP6Signal+0x209)[0x82a1407]
/home/sven/bzr/debug-max/5.1-bugteam/storage/ndb/src/kernel/ndbd(_ZN14SimulatedBlock15executeFunctionEtP6Signal+0xd3)[0x81202f9]
/home/sven/bzr/debug-max/5.1-bugteam/storage/ndb/src/kernel/ndbd(_ZN13FastScheduler5doJobEv+0x1bd)[0x8316001]
/home/sven/bzr/debug-max/5.1-bugteam/storage/ndb/src/kernel/ndbd(_ZN12ThreadConfig13ipControlLoopEv+0x10b)[0x83170c7]
/home/sven/bzr/debug-max/5.1-bugteam/storage/ndb/src/kernel/ndbd(main+0x941)[0x80fa1a5]
/lib/tls/i686/cmov/libc.so.6(__libc_start_main+0xe5)[0xb7d79775]
/home/sven/bzr/debug-max/5.1-bugteam/storage/ndb/src/kernel/ndbd[0x80f8a31]
======= Memory map: ========
08048000-083d2000 r-xp 00000000 08:04 4849860    /home/sven/bzr/debug-max/5.1-bugteam/storage/ndb/src/kernel/ndbd
083d2000-083d9000 r--p 00389000 08:04 4849860    /home/sven/bzr/debug-max/5.1-bugteam/storage/ndb/src/kernel/ndbd
083d9000-084dd000 rw-p 00390000 08:04 4849860    /home/sven/bzr/debug-max/5.1-bugteam/storage/ndb/src/kernel/ndbd
084dd000-0851d000 rw-p 084dd000 00:00 0 
08a88000-08dde000 rw-p 08a88000 00:00 0          [heap]
b0b84000-b127e000 rw-p b0b84000 00:00 0 
b127e000-b127f000 ---p b127e000 00:00 0 
b127f000-b12c3000 rw-p b127f000 00:00 0 
b12c3000-b12c4000 ---p b12c3000 00:00 0 
b12c4000-b1308000 rw-p b12c4000 00:00 0 
b1308000-b1309000 ---p b1308000 00:00 0 
b1309000-b134d000 rw-p b1309000 00:00 0 
b134d000-b134e000 ---p b134d000 00:00 0 
b134e000-b1392000 rw-p b134e000 00:00 0 
b1392000-b1393000 ---p b1392000 00:00 0 
b1393000-b13d7000 rw-p b1393000 00:00 0 
b13d7000-b13d8000 ---p b13d7000 00:00 0 
b13d8000-b141c000 rw-p b13d8000 00:00 0 
b141c000-b141d000 ---p b141c000 00:00 0 
b141d000-b1461000 rw-p b141d000 00:00 0 
b1461000-b1462000 ---p b1461000 00:00 0 
b1462000-b14a6000 rw-p b1462000 00:00 0 
b14a6000-b14a7000 ---p b14a6000 00:00 0 
b14a7000-b14eb000 rw-p b14a7000 00:00 0 
b14eb000-b14ec000 ---p b14eb000 00:00 0 
b14ec000-b1530000 rw-p b14ec000 00:00 0 
b1530000-b1531000 ---p b1530000 00:00 0 
b1531000-b1575000 rw-p b1531000 00:00 0 
b1575000-b1576000 ---p b1575000 00:00 0 
b1576000-b15ba000 rw-p b1576000 00:00 0 
b15ba000-b15bb000 ---p b15ba000 00:00 0 
b15bb000-b15ff000 rw-p b15bb000 00:00 0 
b15ff000-b1600000 ---p b15ff000 00:00 0 
b1600000-b1644000 rw-p b1600000 00:00 0 
b1644000-b1645000 ---p b1644000 00:00 0 
b1645000-b1689000 rw-p b1645000 00:00 0 
b1689000-b168a000 ---p b1689000 00:00 0 
b168a000-b16ce000 rw-p b168a000 00:00 0 
b16ce000-b16cf000 ---p b16ce000 00:00 0 
b16cf000-b1713000 rw-p b16cf000 00:00 0 
b1713000-b1714000 ---p b1713000 00:00 0 
b1714000-b1758000 rw-p b1714000 00:00 0 
b1758000-b1759000 ---p b1758000 00:00 0 
b1759000-b179d000 rw-p b1759000 00:00 0 
b179d000-b179e000 ---p b179d000 00:00 0 
b179e000-b17e2000 rw-p b179e000 00:00 0 
b17e2000-b17e3000 ---p b17e2000 00:00 0 
b17e3000-b1827000 rw-p b17e3000 00:00 0 
b1827000-b1828000 ---p b1827000 00:00 0 
b1828000-b186c000 rw-p b1828000 00:00 0 
b186c000-b186d000 ---p b186c000 00:00 0 
b186d000-b18b1000 rw-p b186d000 00:00 0 
b18b1000-b18b2000 ---p b18b1000 00:00 0 
b18b2000-b18f6000 rw-p b18b2000 00:00 0 
b18f6000-b18f7000 ---p b18f6000 00:00 0 
b18f7000-b193b000 rw-p b18f7000 00:00 0 
b193b000-b193c000 ---p b193b000 00:00 0 
b193c000-b1980000 rw-p b193c000 00:00 0 
b1980000-b1981000 ---p b1980000 00:00 0 
b1981000-b7172000 rw-p b1981000 00:00 0 
b7172000-b7173000 ---p b7172000 00:00 0 
b7173000-b717a000 rw-p b7173000 00:00 0 
b717a000-b717b000 ---p b717a000 00:00 0 
b717b000-b7182000 rw-p b717b000 00:00 0 
b7182000-b7183000 ---p b7182000 00:00 0 
b7183000-b741b000 rw-p b7183000 00:00 0 
b741b000-b7425000 r-xp 00000000 08:02 521254     /lib/tls/i686/cmov/libnss_files-2.9.so
b7425000-b7426000 r--p 00009000 08:02 521254     /lib/tls/i686/cmov/libnss_files-2.9.so
b7426000-b7427000 rw-p 0000a000 08:02 521254     /lib/tls/i686/cmov/libnss_files-2.9.so
b7427000-b7434000 r-xp 00000000 08:02 488685     /lib/libgcc_s.so.1
b7434000-b7435000 r--p 0000c000 08:02 488685     /lib/libgcc_s.so.1
b7435000-b7436000 rw-p 0000d000 08:02 488685     /lib/libgcc_s.so.1
b7436000-b7437000 ---p b7436000 00:00 0 
b7437000-b7d63000 rw-p b7437000 00:00 0 
b7d63000-b7ebf000 r-xp 00000000 08:02 521240     /lib/tls/i686/cmov/libc-2.9.so
b7ebf000-b7ec0000 ---p 0015c000 08:02 521240     /lib/tls/i686/cmov/libc-2.9.so
b7ec0000-b7ec2000 r--p 0015c000 08:02 521240     /lib/tls/i686/cmov/libc-2.9.so
b7ec2000-b7ec3000 rw-p 0015e000 08:02 521240     /lib/tls/i686/cmov/libc-2.9.so
b7ec3000-b7ec6000 rw-p b7ec3000 00:00 0 
b7ec6000-b7eea000 r-xp 00000000 08:02 521249     /lib/tls/i686/cmov/libm-2.9.so
b7eea000-b7eeb000 r--p 00023000 08:02 521249     /lib/tls/i686/cmov/libm-2.9.so
b7eeb000-b7eec000 rw-p 00024000 08:02 521249     /lib/tls/i686/cmov/libm-2.9.so
b7eec000-b7f01000 r-xp 00000000 08:02 521251     /lib/tls/i686/cmov/libnsl-2.9.so
b7f01000-b7f02000 r--p 00014000 08:02 521251     /lib/tls/i686/cmov/libnsl-2.9.so
b7f02000-b7f03000 rw-p 00015000 08:02 521251     /lib/tls/i686/cmov/libnsl-2.9.so
b7f03000-b7f05000 rw-p b7f03000 00:00 0 
b7f05000-b7f0e000 r-xp 00000000 08:02 521247     /lib/tls/i686/cmov/libcrypt-2.9.so
b7f0e000-b7f0f000 r--p 00008000 08:02 521247     /lib/tls/i686/cmov/libcrypt-2.9.so
b7f0f000-b7f10000 rw-p 00009000 08:02 521247     /lib/tls/i686/cmov/libcrypt-2.9.so
b7f10000-b7f38000 rw-p b7f10000 00:00 0 
b7f38000-b7f4d000 r-xp 00000000 08:02 521259     /lib/tls/i686/cmov/libpthread-2.9.so
b7f4d000-b7f4e000 r--p 00014000 08:02 521259     /lib/tls/i686/cmov/libpthread-2.9.so
b7f4e000-b7f4f000 rw-p 00015000 08:02 521259     /lib/tls/i686/cmov/libpthread-2.9.so
b7f4f000-b7f51000 rw-p b7f4f000 00:00 0 
b7f62000-b7f67000 rw-p b7f62000 00:00 0 
b7f67000-b7f68000 r-xp b7f67000 00:00 0          [vdso]
b7f68000-b7f84000 r-xp 00000000 08:02 488671     /lib/ld-2.9.so
b7f84000-b7f85000 r--p 0001b000 08:02 488671     /lib/ld-2.9.so
b7f85000-b7f86000 rw-p 0001c000 08:02 488671     /lib/ld-2.9.so
bf96f000-bf986000 rw-p bffe9000 00:00 0          [stack]
090702 14:48:23 [Note] Plugin 'FEDERATED' is disabled.
090702 14:48:23 [Note] Plugin 'ndbcluster' is disabled.
090702 14:48:23 [Warning] Forcing shutdown of 2 plugins

How to repeat:
I can reproduce this maybe 2-10% of the times, using the following test case:

source include/have_ndb.inc;
create table t1 (a int) engine = ndb;
insert into t1 values (1);
drop table t1;
exit;

There is nothing special with this test case, I have seen it when running other tests too.

I execute the test case as follows:

cd mysql-test
n=0
while [ 1 ] ; do
  n=$((n+1))
  echo ==== $n ====
  ./mtr ndb_stack_smashing
done

So far it has failed 3 times out of 38.
[14 Sep 2009 6:50] Sveta Smirnova
Thank you for the report.

I can not repeat described behavior with current 5.1-bugteam tree. Please try it now and if you still able to repeat answer Jørgen's questions
[14 Sep 2009 13:06] Sven Sandberg
I can still repeat this. I pulled today's 5.1-bugteam, revid:luis.soares@sun.com-20090913214347-nvzyipdt40e21rx7 and ran the same test with similar output.

I have built using ./BUILD/compile-pentium-debug-max .

I'm using Ubuntu Jaunty.

$ uname -a
Linux riska 2.6.28-15-generic #49-Ubuntu SMP Tue Aug 18 18:40:08 UTC 2009 i686 GNU/Linux

Let me know if you need anything else!