MySQL Bugs: #28515: Falcon falcon_bug_22154 hangs on Linux 64-bit.

Bug #28515	Falcon falcon_bug_22154 hangs on Linux 64-bit.
Submitted:	18 May 2007 10:40	Modified:	2 Jul 2007 4:52
Reporter:	Hakan Küçükyılmaz	Email Updates:
Status:	Closed	Impact on me:	None
Category:	MySQL Server: Falcon storage engine	Severity:	S1 (Critical)
Version:	6.0.1-alpha	OS:	Linux (64-bit)
Assigned to:	Jim Starkey	CPU Architecture:	Any

Description:
falcon_bug_22154 hangs on Linux 64-bit.

How to repeat:
./mysql-test-run.pl --enable-disabled --testcase-timeout=600 falcon_bug_22154

I stopped the test after 2.5 hours run time. It hangs around IndexRootPage:594

for (Bdb *bdb = findInsertionLeaf (dbb, indexId, &searchKey, recordNumber,
transId); bdb;
		 bdb = dbb->handoffPage (bdb, page->nextPage, PAGE_btree, Exclusive))

I see constant disk write activity and recordNumber is growing steady. While
debugging recordNumber was 109708.

While running falcon_bug_22154 in debugger I get alot of following error messages

Index deletion failed for record 25801.0 of SYSTEM.FIELDS.FIELDS..PRIMARY_KEY
Index deletion failed for record 25803.0 of SYSTEM.FIELDS.FIELDS..PRIMARY_KEY
Index deletion failed for record 25804.0 of SYSTEM.FIELDS.FIELDS..PRIMARY_KEY
Index deletion failed for record 25805.0 of SYSTEM.FIELDS.FIELDS..PRIMARY_KEY
Index deletion failed for record 25806.0 of SYSTEM.FIELDS.FIELDS..PRIMARY_KEY
Index deletion failed for record 25808.0 of SYSTEM.FIELDS.FIELDS..PRIMARY_KEY

After some time I get following crash
Index deletion failed for record 125301.0 of SYSTEM.FIELDS.FIELDS..PRIMARY_KEY
Index deletion failed for record 125316.0 of SYSTEM.FIELDS.FIELDS..PRIMARY_KEY
Index deletion failed for record 125348.0 of SYSTEM.FIELDS.FIELDS..PRIMARY_KEY
 18835192 bytes reclaimed
Transactions: 770 committed, 0 rolled back, 1 active, 1 post-commit
Activity on falcon_master.fts: 4334168 fetches, 8176 reads, 9668 writes
Activity on /dev/shm/var/master-data/falcon_user.fts: 367442 fetches, 49578 reads, 51891 writes
SerialLog: 64 reads, 1140 writes, 87 transactions, 414 blocks, 43 windows
Switching log files (75303936 used)
Record Memory usage for falcon_master:
  5. 46029432
  4. 230378508
  3. 185
  2. 147
  1. 112
 total: 276408384, threshold 5 -- scavenge
 259707192 bytes reclaimed
Transactions: 8 committed, 0 rolled back, 1 active, 0 post-commit
Activity on falcon_master.fts: 0 fetches, 0 reads, 3 writes
Activity on /dev/shm/var/master-data/falcon_user.fts: 35928 fetches, 12012 reads, 12097 writes
SerialLog: 0 reads, 43 writes, 8 transactions, 47 blocks, 6 windows
Exception: serial write error on "falcon_master.fl1": No such file or directory (2)
terminate called after throwing an instance of 'SQLError'

Program received signal SIGABRT, Aborted.
0x00002b63c8a88c45 in ?? ()
(gdb) bt
#0  0x00002b63c8a88c45 in ?? ()
#1  0x00002b63c8a8a5c0 in ?? ()
#2  0x00000000000000e1 in ?? ()
#3  0x00002b63c7914824 in ?? ()
#4  0x00002b63c7435f68 in ?? ()
#5  0x000000004007d1e0 in ?? ()
#6  0x0000000000a80b27 in typeinfo name for SQLException ()
#7  0x0000000000a80b30 in typeinfo name for SQLError ()
#8  0x0000000000000011 in ?? ()
#9  0x000000004007d020 in ?? ()
#10 0x0000001200000001 in ?? ()
#11 0x00002b63c8a5e790 in ?? ()
#12 0x00002b63c85c7960 in ?? ()
#13 0x0000000000000000 in ?? ()
(gdb)

Thank you for the bug report. I was not able to repeat on Suse 10.1 64-bit.
Could you please test with latest source. Thanks in advance:

Starting Tests in the 'main' suite

Test(s) which will be run though they are marked as disabled:
  falcon_bug_22154     : Bug#22154 2006-11-04 hakank Too long run time for PB

TEST                           RESULT         TIME (ms)
-------------------------------------------------------

falcon_bug_22154               [ pass ]         717781
-------------------------------------------------------
Stopping All Servers
All 1 tests were successful.
The servers were restarted 1 times
Spent 717.781 seconds actually executing testcases

miguel@luar:~/dbs/mysql-5.1-falcon/mysql-test> file ../sql
sql/        sql-bench/  sql-common/
miguel@luar:~/dbs/mysql-5.1-falcon/mysql-test> file ../sql/mysqld
../sql/mysqld: ELF 64-bit LSB executable, AMD x86-64, version 1 (SYSV), for GNU/Linux 2.6.4, dynamically linked (uses shared libs), for GNU/Linux 2.6.4, not stripped
miguel@luar:~/dbs/mysql-5.1-falcon/mysql-test>

It does not hang for me neither anymore:

TEST                           RESULT         TIME (ms)
-------------------------------------------------------

falcon_bug_22154               [ pass ]         437554
-------------------------------------------------------
Stopping All Servers
All 1 tests were successful.
The servers were restarted 1 times
Spent 437.554 seconds actually executing testcases

A note has been added to the 6.0.1 changelog: 

mysqld would crash after a high number of ALTER TABLE, INSERT and UPDATEstatements. (Bug#22154, Bug#28515)