MySQL Bugs: #33517: Falcon crash on recovery

Bug #33517	Falcon crash on recovery
Submitted:	26 Dec 2007 15:10	Modified:	9 Oct 2008 11:24
Reporter:	Philip Stoev	Email Updates:
Status:	Closed	Impact on me:	None
Category:	MySQL Server: Falcon storage engine	Severity:	S3 (Non-critical)
Version:	6.0.4	OS:	Any
Assigned to:	Jim Starkey	CPU Architecture:	Any

Description:
Falcon table recovery crashes repeatedly after starting the server on a corrupted tablespace.

The tablespace was corrupted by running the iuds6 systems test against Falcon, which resulted in server hanging and no new queries or clients were accepted. After the server was killed with kill -9, the server crashed repeatedly when trying to access a Falcon table.

The stack trace is as follows:

0x824a171 handle_segfault + 441
0x4003ace6 _end + 929182550
0x83fcc8d _ZN5Error5errorEPKcz + 73
0x83fcd96 _ZN5Error15assertionFailedEPKci + 22
0x83c411b _ZN10SyncObjectD1Ev + 83
0x8472797 _ZN5CacheD0Ev + 83
0x83ebfc8 _ZN8DatabaseD0Ev + 796
0x83e8ea9 _ZN10Connection11getDatabaseEPKcS1_P7Threads + 365
0x83e6712 _ZN10Connection12openDatabaseEPKcS1_S1_S1_S1_P7Threads + 74
0x83bb25b _ZN15StorageDatabase17getOpenConnectionEv + 63
0x83befd8 _ZN14StorageHandler10initializeEv + 108
0x83bea42 _ZN14StorageHandler20getStorageConnectionEP17StorageTableShareP3THDi10OpenOption + 426
0x83b1452 _ZN16StorageInterface4openEPKcij + 98
0x830aac2 _ZN7handler7ha_openEP8st_tablePKcii + 34
0x831281d _ZN12ha_partition4openEPKcij + 405
0x830aac2 _ZN7handler7ha_openEP8st_tablePKcii + 34
0x828b38b _Z21open_table_from_shareP3THDP14st_table_sharePKcjjjP8st_table15open_table_mode + 783
0x82885f0 _Z17open_unireg_entryP3THDP8st_tableP10TABLE_LISTPKcPcjP11st_mem_rootj + 124
0x8283152 _Z10open_tableP3THDP10TABLE_LISTP11st_mem_rootPbj + 1954

How to repeat:
1. Download the zipfile containing the corrupted tablespace and unpack it.

2. Run the 6.0.4 server against that tablespace

3. Open a mysql client and issue a "use systest1"

4. At that point Falcon will engage in I/O and then crash predictably.

Tablespace archive is at ftp://ftp.mysql.com/pub/mysql/upload/bug33517.zip

Call Stack on Windows

Attachment: call_stack_windows_bug33517.txt (text/plain), 4.37 KiB.

Thank you for the bug report.

c:\dbs>6.0t\bin\mysql -uroot
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 1
Server version: 6.0.5-alpha-team-tree-nt Source distribution

Type 'help;' or '\h' for help. Type '\c' to clear the buffer.

mysql> use systest1
Database changed
mysql> show tables;
+--------------------+
| Tables_in_systest1 |
+--------------------+
| ddl_logs           |
| event_log          |
| tb0_eng1           |
| tb0_eng2           |
| tb0_logs           |
| tb0_master         |
| tb1_aux            |
| tb1_eng1           |
| tb1_eng2           |
| tb1_logs           |
| tb1_master         |
| tb2_eng1           |
| tb3_eng1           |
| tb4_eng1           |
| tb5_eng1           |
| tb6_eng1           |
| tb7_eng1           |
| tb8_eng1           |
| tb9_eng1           |
| test_stat          |
+--------------------+
20 rows in set (0.19 sec)

mysql> desc tb0_eng1\G
ERROR 2013 (HY000): Lost connection to MySQL server during query
mysql>

See Call Stack attached.

This bug is about the crash in the recovery, about the actual crash which lead to the need to recover, please see bug #33518.

Thank you for doing the stack so prettily -- I will take care to do that myself in the future.

Running recovery with falcon_debug_log=65535 causes this to be printed into the log before the crash:

Recovering database /home/qauser/6.0.4/systest_vardir-hang/master-data/falcon_master.fts ...
first recovery block is 3806
last recovery block is 62295
recovery read block is 6365
Exception: buffer pool is exhausted

Bugcheck: assertion failed at line 125 in file SyncObject.cpp

071227  9:50:17 - mysqld got signal 4;

I was able to unzip and recover the database files in the archive 
shell.mysql.com:/supportftp/pub/mysql/upload/bug33517.zip without error.

The dates of the files in the archive correspond to the description above, but there are no table spaces other than the defaults, and only a single table, T1, present.

Is it possible that the archive does not correspond to the original bug?

If the problem can be made to re-appear, please re-open the bug.  But for the time being, there's nothing more I can do with it.

I am able to repeat the bug with the latest BK tree. Here are the steps I used:

1. compile 6.0 BK with compile-pentium-debug-max-no-ndb
2. Run mysqld:

./mysqld --datadir=/build/bug33517/master-data/ --language=/build/mysql-6.0/sql/share/english --falcon_debug_mask=65535

3. With a mysql client, execute:

mysql> use systest1;
Database changed
mysql> show tables;
mysql> desc tb0_eng1;
ERROR 2013 (HY000): Lost connection to MySQL server during query

The mysqld stderr contains numerous instances of:

"DataPage::computeSpaceAvailable got a negative number"

Linux philips 2.6.21-1.3194.fc7 #1 SMP Wed May 23 22:35:01 EDT 2007 i686 i686 i386 GNU/Linux

32-bit virtual machine inside 32-bit VMware Player on an a 32-bit OS on a Intel Core Duo.

I can confirm that

shell.mysql.com:/supportftp/pub/mysql/upload/bug33517.zip 

is the correct file to reproduce the problem. This tablespace does not contain a table named T1. Instead, it contains a systest1 database with contents as described by Mr. Solorzano.

There was a missing call to release a buffer descriptor block in a recovery path.

Patch is in mysql-6.0-release version 6.0.4

Documented bugfix in the 6.0.4 changelog as follows:

        Table recovery failed repeatedly after starting the server with a
        corrupted Falcon tablespace, which caused the server to crash.