Bug #43765 falcon_bug_22173a crashes in IndexPage::findNodeInLeaf
Submitted: 20 Mar 2009 9:51 Modified: 15 May 2009 15:51
Reporter: Hakan Küçükyılmaz Email Updates:
Status: Closed Impact on me:
None 
Category:MySQL Server: Falcon storage engine Severity:S1 (Critical)
Version:6.0-falcon-team OS:Linux
Assigned to: Vladislav Vaintroub
Tags: F_INDEX, pushbuild, test failure
Triage: Triaged: D1 (Critical)

[20 Mar 2009 9:51] Hakan Küçükyılmaz
Description:
falcon_bug_22173a fails 1 out of 100 times

How to repeat:
Run falcon_bug_22173a several times. I tried it running 112 times

falcon.falcon_bug_22173a                 [ fail ]
        Test ended at 2009-03-20 10:09:07

CURRENT_TEST: falcon.falcon_bug_22173a
mysqltest: At line 56: query 'call p1()' failed: 2013: Lost connection to MySQL server during query

The result from queries just before the failure was:
< snip >
declare continue handler for 1020 begin end;
declare continue handler for 1213 begin end;
declare continue handler for 1015 begin end;
while v1 < 2500 do
/* SELECT 'insert', v1; */
INSERT INTO t1 VALUES (v1);
SET v2 = rand() * 10000;
UPDATE t1 SET a = v2 WHERE a = v1;
/* SELECT ' update', v1; */
SET v1 = v1 + 1;
if v1 mod 50 = 0 then
/* SELECT ' delete'; */
/* DELETE FROM t1; */
TRUNCATE t1;
end if;
end while;
end//
# Sent call p1() to the server but do not pull the results
call p1();
# Switch to connection conn1

More results from queries before failure can be found in /home/hakan/work/mysql/mysql-6.0-falcon-team/mysql-test/var/log/falcon_bug_22173a.log

 - saving '/home/hakan/work/mysql/mysql-6.0-falcon-team/mysql-test/var/log/falcon.falcon_bug_22173a/' to '/home/hakan/work/mysql/mysql-6.0-falcon-team/mysql-test/var/log/falcon.falcon_bug_22173a/'
 - found 'core.26578' (0/5)

Trying 'dbx' to get a backtrace

Trying 'gdb' to get a backtrace
Core generated by '/home/hakan/work/mysql/mysql-6.0-falcon-team/sql/mysqld'
Output from gdb follows. The first stack trace is from the failing thread.
The following stack traces are from all threads (so the failing one is
duplicated).
--------------------------

warning: Can't read pathname for load map: Input/output error.
Core was generated by `/home/hakan/work/mysql/mysql-6.0-falcon-team/sql/mysqld --defaults-group-suffix'.
Program terminated with signal 6, Aborted.
[New process 26590]
[New process 32572]
[New process 32571]
[New process 26594]
[New process 26593]
[New process 26592]
[New process 26591]
[New process 26578]
[New process 26589]
[New process 26588]
[New process 26587]
[New process 26586]
[New process 26585]
[New process 26584]
[New process 26583]
[New process 26582]
[New process 26581]
[New process 26580]
#0  0x00007fb5e72e5b16 in pthread_kill () from /lib/libpthread.so.0
#0  0x00007fb5e72e5b16 in pthread_kill () from /lib/libpthread.so.0
#1  0x0000000000683a4a in handle_segfault (sig=6) at mysqld.cc:2689
#2  <signal handler called>
#3  0x00007fb5e72e866b in raise () from /lib/libpthread.so.0
#4  0x00000000008e2af8 in Error::error (string=<value optimized out>) at Error.cpp:94
#5  0x0000000000969fe4 in IndexPage::findNodeInLeaf (this=0x7fb5e54f1000, indexKey=0x7fb5dabf2bc0,
    foundKey=0x0) at IndexPage.cpp:199
#6  0x00000000008f110a in IndexRootPage::addIndexEntry (dbb=0x7fb5e594c790, indexId=1, key=0x7fb5dabf2bc0,
    recordNumber=408, transId=0) at IndexRootPage.cpp:138
#7  0x00000000008f1c80 in IndexRootPage::indexMerge (dbb=0x7fb5e594c790, indexId=1, logRecord=0x7fb5dabf5d88,
    transId=0) at IndexRootPage.cpp:887
#8  0x000000000094a193 in SRLUpdateIndex::commit (this=0x7fb5dabf5d88) at SRLUpdateIndex.cpp:195
#9  0x00000000009318a1 in SerialLogTransaction::commit (this=0x4024e68) at SerialLogTransaction.cpp:92
#10 0x0000000000964d75 in Gopher::gopherThread (this=0x7fb5e558bb50) at Gopher.cpp:71
#11 0x00000000008a80f3 in Thread::thread (this=0x7fb5e5595668) at Thread.cpp:166
#12 0x00000000008a82c1 in Thread::thread (parameter=0x67d2) at Thread.cpp:145
#13 0x00007fb5e72e0faa in start_thread () from /lib/libpthread.so.0
#14 0x00007fb5e625b2bd in clone () from /lib/libc.so.6
#15 0x0000000000000000 in ?? ()
[20 Mar 2009 9:52] Hakan Küçükyılmaz
Full stack trace

Attachment: stack.trace (application/octet-stream, text), 28.48 KiB.

[20 Mar 2009 15:30] Hakan Küçükyılmaz
Verified by Pushbuild.
[23 Mar 2009 12:28] Kevin Lewis
See also the duplicate Bug#43800
[23 Mar 2009 22:14] John Embretsen
For what it is worth, last weekend I ran this test 3000 times on two different Solaris hosts (with no/little background load and plenty of memory) without any failures.
[1 Apr 2009 21:30] Kevin Lewis
See also Duplicate Bug#43954
[9 Apr 2009 7:07] Bugs System
A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/71722

3110 Vladislav Vaintroub	2009-04-09
      Bug #43765 falcon_bug_22173a crashes in IndexPage::findNodeInLeaf
      
      Falcon crashes with assertion (level==0)in high-concurrent
      index updates
      
      Reason is that recent code refactoring in IndexPage code introduced a 
      regression.
      
      When index is small and consists of a single root page only, findInsertionLeaf(), would check lock the root page with shared lock, check if the root page level is 0 (page is a leaf),  unlock the page, lock it again excusively and return this page. The problem is that during the gap while page was not locked, it could have beens spli. In this case returning root page is wrong, as it is not a "insertion leaf", and generally  not leaf page at all. 
      
      The fix is not to return from the function prematurely, when reacquiring lock in the root and root page changes the level from 0 to greater number.
[9 Apr 2009 13:46] Kevin Lewis
Code looks correct now  OK to push.
[15 Apr 2009 17:00] Bugs System
Pushed into 6.0.11-alpha (revid:hky@sun.com-20090415164923-9arx29ye5pzxd4zf) (version source revid:vvaintroub@mysql.com-20090409070645-katx7bwydr9y6ftt) (merge vers: 6.0.11-alpha) (pib:6)
[15 May 2009 15:51] MC Brown
An entry has been added to the 6.0.11 changelog: 

When performing a high number of concurrent index updates on a Falcon table, mysqld could crash due to an assertion.