Bug #39951 Falcon: crash during crash recovery
Submitted: 9 Oct 2008 11:23 Modified: 9 Jan 2009 14:09
Reporter: Alexey Stroganov Email Updates:
Status: Closed Impact on me:
None 
Category:MySQL Server: Falcon storage engine Severity:S1 (Critical)
Version:6.0.7, 6.0-falcon.06.10.08 OS:Any
Assigned to: Vladislav Vaintroub CPU Architecture:Any
Tags: F_RECOVERY
Triage: Triaged: D1 (Critical)

[9 Oct 2008 11:23] Alexey Stroganov
Description:
After loading of flightstat database and creation of indices I shutdown the server and start it again. Server crashed right away on the very beginning stage.

(gdb) bt
#0  0x00002b319b3b94c5 in pthread_kill () from /lib64/libpthread.so.0
#1  0x0000000000a57098 in my_write_core (sig=11) at stacktrace.c:307
#2  0x0000000000631163 in handle_segfault (sig=11) at mysqld.cc:2660
#3  <signal handler called>
#4  Table::insert (this=0xeeeeeeeeeeeeeeee, record=0x0, prior=0x2aaacb0beb40, recordNumber=30) at Table.cpp:1929
#5  0x000000000083f1c0 in Table::rollbackRecord (this=0xeeeeeeeeeeeeeeee, recordToRollback=0x2aaacb0beb40,
    transaction=0x2aaaab454170) at Table.cpp:1010
#6  0x0000000000892f10 in RecordVersion::rollback (this=0x1e, transaction=<value optimized out>) at RecordVersion.cpp:162
#7  0x0000000000845550 in Transaction::rollback (this=0x2aaaab454170) at Transaction.cpp:424
#8  0x0000000000857f68 in Connection::rollback (this=0x2aaaab4ae7a0) at Connection.cpp:282
#9  0x000000000085ddf4 in ~Database (this=0x2aaaab2acbc8) at Database.cpp:553
#10 0x000000000085646b in Connection::getDatabase (this=0x2aaaab4ad7f8, dbName=0x2aaaab4ad364 "FALCON_MASTER",
    dbFileName=0x7fff0f815a50 "/data1/mysql/flightstats_test/falcon_master.fts", threads=0x2aaaab4ad3d0)
    at Connection.cpp:1657
#11 0x00000000008579f2 in Connection::openDatabase (this=0x2aaaab4ad7f8, dbName=0x2aaaab4ad364 "FALCON_MASTER",
    filename=<value optimized out>, account=0xb48e68 "mysql", password=0xb48e68 "mysql", address=0x0, parent=0x2aaaab4ad3d0)
    at Connection.cpp:934
#12 0x000000000082bc9b in StorageDatabase::getOpenConnection (this=0x2aaaab4ad1d0) at StorageDatabase.cpp:136
#13 0x000000000082d95b in StorageHandler::initialize (this=0x2aaaab2ac048) at StorageHandler.cpp:978
#14 0x000000000082188c in StorageInterface::falcon_init (p=0x1a3a3b0) at ha_falcon.cpp:215
#15 0x000000000072080b in ha_initialize_handlerton (plugin=0x1a31138) at handler.cc:426
#16 0x00000000007b53bd in plugin_initialize (plugin=0x1a31138) at sql_plugin.cc:1011
#17 0x00000000007b774c in plugin_init (argc=0xf8ed10, argv=0x1a00c58, flags=<value optimized out>) at sql_plugin.cc:1218
#18 0x0000000000633b6d in init_server_components () at mysqld.cc:4045
#19 0x000000000063442c in main (argc=<value optimized out>, argv=<value optimized out>) at mysqld.cc:4492

This crash observed with 6.0.7 and 6.0.8pre

How to repeat:
Try to server with this flightstat database. I will post details how to access this database separately.
[13 Oct 2008 21:03] Bugs System
A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/56151

2862 Vladislav Vaintroub	2008-10-13
      Bug#39951 Falcon: crash during crash recovery.
      
      The problem is that in Database destructor, tables are accessed
      deep in systemConnection->rollback(), at the time all Table objects
      have been freed. Crash reason is accessing freed memory.
      
      Solution is to cleanup system connection before Table objects are
      freed.
[13 Oct 2008 21:46] Vladislav Vaintroub
This particular patch fixes only this particular crash. Anything else (like the rest of recovery problems) is not subject of this patch.
[28 Oct 2008 8:10] Bugs System
Pushed into 6.0.8-alpha  (revid:vvaintroub@mysql.com-20081013210224-soy6yyyud2bq6wol) (version source revid:vvaintroub@mysql.com-20081013210224-soy6yyyud2bq6wol) (pib:5)
[9 Jan 2009 14:09] MC Brown
A note has been added to the 6.0.8 changelog: 

MySQL may crash during the recover of Falcon tables if the server was shutdown after a large data load.