Bug #33064 Shutdown fails if Falcon hangs in a test case
Submitted: 7 Dec 2007 14:01 Modified: 16 Mar 2008 10:52
Reporter: Hakan Küçükyılmaz Email Updates:
Status: Closed Impact on me:
None 
Category:MySQL Server: Falcon storage engine Severity:S1 (Critical)
Version:mysql-6.0-falcon T1.0-4 OS:Any
Assigned to: Christopher Powers CPU Architecture:Any

[7 Dec 2007 14:01] Hakan Küçükyılmaz
Description:
Shutdown fails if Falcon hangs in a test case.

How to repeat:
./mysql-test-run.pl --enable-disabled --force --suite=falcon falcon_bug_22154

Test(s) which will be run though they are marked as disabled:
  falcon.falcon_bug_22154 : Bug#22154 2006-11-04 hakank Too long run time for PB

TEST                           RESULT         TIME (ms)
-------------------------------------------------------

falcon.falcon_bug_22154        [ fail ]  timeout

Stopping All Servers

-- It hangs here

Please note that falcon_bug_22154 hangs in Falcon and get's a timeout from mysql-test-run.pl. The shutdown works in mysql-6.0 main tree, but not in mysql-6.0-falcon tree.

I will attach the backtrace of the hanging situation provided by Christoffer Hall.

Suggested fix:
Don't hang.
[7 Dec 2007 14:03] Hakan Küçükyılmaz
Backtrace of hang

Attachment: bug_33064_backtrace.txt (application/unknown, text), 12.71 KiB.

[7 Dec 2007 14:41] Kevin Lewis
Hakan, Please confirm that this hang occurs on the mysql-6.0-falcon that was updated Wednesday, Dec 5, by Chris for Bug#32963.  The reason I ask is that this change;

ChangeSet@1.2732.2.1, 2007-12-04 20:46:39-06:00, chris@xeno.mysql.com +2 -0
  Bug#32963, falcon_bug_22173a fails
  - Eliminated potential deadlock between scavenger and truncate operation

re-ordered the succession of locks in Database::updateCardinalities() so that
Database::syncTables is taken before Database::syncSysConnection.

In the attached stack trace, one thread is in Database::updateCardinalities() waiting on Database::syncSysConnection, and the other thread is in;
 
StorageDatabase::renameTable()->Table::rename()->Database::renameTable()

and waiting on Database::syncTables.  Unfortunately, this thread already got a lock on Database::syncSysConnection back up in StorageDatabase::renameTable().

The possibility for this deadlock was introduced Wednesday when updateCardinalities was changed and is not in the 6.0.4 clone-off.

You can add the information_schema.falcon_version where this was found.
[7 Dec 2007 14:45] Christoffer Hall
Hi Kevin

The code is mysql-6.0-falcon as of a couple of hours ago.

Cheers
C
[7 Dec 2007 15:09] Hakan Küçükyılmaz
With change set ChangeSet@1.2732, 2007-12-03 17:28:37+01:00, hakank@lu0011.(none) +1 -0

it does not hang.

TEST                           RESULT         TIME (ms)
-------------------------------------------------------

falcon.falcon_bug_22154        [ fail ]

ERROR: Bug#22154 2006-11-04 hakank Too long run time for PB

Stopping All Servers
Restoring snapshot of databases
Resuming Tests

-------------------------------------------------------
Stopping All Servers
Failed 1/1 tests, 0.00% were successful.
[7 Dec 2007 15:21] Kevin Lewis
Hakan, this is a deadlock between the foreground thread doing renameTable and the scavenger thread, which starts up every 30 seconds.  It is very timing dependent and intermittent.
[7 Dec 2007 17:02] MySQL Verification Team
Thank you for the bug report.

TEST                           RESULT         TIME (ms)
-------------------------------------------------------

falcon.falcon_bug_22154        [ fail ]  timeout

Stopping All Servers
[11 Dec 2007 15:28] Hakan Küçükyılmaz
A stored procedure to check whether steady meta data changes is revealing any memory leak in Falcon hangs also after 30 seconds run time. The stored procedure is

   DROP SCHEMA test;
   CREATE SCHEMA test;
   USE test;
   
   SET @@sql_mode = 'ANSI_QUOTES';
   
   DELIMITER //
   CREATE PROCEDURE p1()
   BEGIN
       CREATE TABLE t1 (a int, b varchar(50), PRIMARY KEY (a), KEY (b)) Engine Falcon;
       CREATE TABLE t2 (a int) Engine Falcon;
   
       SET @table_name = 't1';
       SET @field_name = 'a';
   
       SET @i = 0;
       WHILE @i < 1000000 DO
           SET @new_table_name = md5(@table_name);
           SET @new_field_name = md5(@field_name);
   
           SET @stmt1 = concat('RENAME TABLE "', @table_name, '" TO "', @new_table_name, '"');
           SET @stmt2 = concat('ALTER TABLE "t2" CHANGE "', @field_name, '" "', @new_field_name, '" integer');
   
           -- SELECT @stmt1;
           -- SELECT @stmt2;
   
           PREPARE stmt1 FROM @stmt1;
           PREPARE stmt2 FROM @stmt2;
   
           EXECUTE stmt1;
           EXECUTE stmt2;
   
           SET @table_name = @new_table_name;
           SET @field_name = @new_field_name;
           SET @i = @i + 1;
       END WHILE;
   END;//
   
   DELIMITER ;
   CALL p1;
[13 Dec 2007 4:03] Kevin Lewis
Pushed changeset 1.2752 to mysql-6.0-falcon-team
The order that should be followed is Database:syncTables,
followed by Table::syncObject, then Database::syncSysConnection
[21 Feb 2008 16:18] Kevin Lewis
Patch is in mysql-6.0-release version 6.0.4
[12 Mar 2008 23:03] Bugs System
Pushed into 6.0.4-alpha
[16 Mar 2008 10:52] MC Brown
Test failure - no changelog entry required.