Bug #39138 Falcon total database deletion after CREATE + DROP TABLESPACE + recovery
Submitted: 30 Aug 2008 20:18 Modified: 4 Oct 2008 14:54
Reporter: Philip Stoev Email Updates:
Status: Closed Impact on me:
None 
Category:MySQL Server: Falcon storage engine Severity:S1 (Critical)
Version:6.0-falcon-team OS:Any
Assigned to: Vladislav Vaintroub CPU Architecture:Any

[30 Aug 2008 20:18] Philip Stoev
Description:
When Falcon recovers from a serial log file containing a lot of CREATE TABLESPACE and DROP TABLESPACE, the following situation occurs:

New table space FALCON_USER, id 1, type 0, filename falcon_user.fts
New table space FALCON_TEMPORARY, id 2, type 0, filename falcon_temporary.fts
Recovering database /build/bzr/6.0-falcon-team/mysql-test/var/master-data/falcon_master.fts ...
first recovery block is 1
last recovery block is 1309
recovery read block is 45
Exception: can't open file "f3": No such file or directory (2)
Couldn't open table space file "f3" for tablespace "c": can't open file "f3": No such file or directory (2)
Exception: can't open file "f3": No such file or directory (2)
IO::createFile: creating file "falcon_master.fts"

At this point the original falcon_master.fts has been obliterated. All data from all tablespaces has been lost.

How to repeat:
A test case will be provided shortly.

Suggested fix:
This is wrong on so many levels:

* IO::createFile should never ever create a file over a file that already exists, no matter what.

* The Recovery should take into account that some tablespaces may have been both created and dropped in the duration of the serial log. It should not try to open tablespace files that only existed temporarily and were later dropped. To achieve that, I assume that one of the initial recovery passes over the serial log should keep track which tablespace files remain valid throughout and which were dropped.

* An Exception during Falcon recovery should cause the server to stop. Instead, the server continues and starts accepting connections as if nothing has happened.

* Falcon files should only be created in groups of three (the falcon_master tablespace and the two serial log files) and under no circumstances should just one of those files be created while leaving the rest in an unknown state. Any time the files are not present together should result in a hard failure and server termination. In this particular bug, the serial log files were left intact while the tablespace file was recreated. This is a No, No, No (Amy Winehouse style).
[30 Aug 2008 20:25] Philip Stoev
Grammar file for bug 39138

Attachment: bug39138.yy (application/octet-stream, text), 581 bytes.

[30 Aug 2008 20:32] Philip Stoev
To reproduce this bug, please clone the latest revision of mysql-test-extra-6.0 and execute:

$ cd mysql-test/gentest
$ perl runall.pll \
  --basedir=/path/to/6.0-falcon-team \
  --grammar=conf/bug39138.yy \
  --reporters=Recovery \
  --threads=1 \
  --queries=1000

This will proceed to execute 1000 CREATE/DROP TABLESPACE/TABLE queries. After that, it will kill the server and initiate recovery. The output will show Falcon exceptions and the tablespace will be recreated. If you start the server again, you will see that SHOW TABLES continues to list varous tables, but they are all unreadable, regardless of what tablespace they were defined in.
[4 Sep 2008 10:50] Bugs System
A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/53247

2808 Vladislav Vaintroub	2008-09-04
      Bug#39138 : Falcon total database deletion after CREATE + DROP TABLESPACE + recovery
      
      Problem: Falcon misinterpretins some errors from openDatabase(), assumes database does 
      not exist and tries to create a new database. During creation, it  overwrites already existing 
      files (by passing O_CREAT|O_TRUNC to open() on Posix, and CREATE_ALWAYS to CreateFile on Windows).
      
      This is corrected in this patch 
      
      - open() is now called with O_CREAT|O_EXCL  and CreateFile() with CREATE_NEW. 
      This will make Falcon never overwrite existing files.
      
      - To ensure the proper cleanup of files created during create database attempt, 
      "create" status is  tracked. On error, createDatabase sets global deleteFilesOnExit 
      flag - this causes new files to be deleted in destructors of IO and SerialLogFile. 
      But there is no cleanup if process terminates with exception/signal.
      A more elegant solution for cleanup is possible on file systems that support user-mode 
      transactions (like recent NTFS), by making the whole createDatabase/openDatabase a single 
      transaction with commit() at the end.  Unfortunately this solution is not portable yet,
      not even on downlevel  Windows ,so it is not implemented.
      
      - We also ensure with ASSERT, that createDatabase never opens existing files, but always 
      creates new ones.
      - We do not attempt to create a database after failing recovery anymore.
[10 Sep 2008 17:15] Kevin Lewis
I have reviewed this patch and it has been pushed. Philip, can you verify that this fix is complete?
[1 Oct 2008 19:53] Philip Stoev
The biggest recovery issue mentioned in this bug -- the complete deletion of all tablespaces, has been resolved by this patch. Upon Vlad's request, a separate bug will be filed to track the rest of the recovery issues involving tablespaces.
[1 Oct 2008 20:53] Philip Stoev
The new bug is 

http://bugs.mysql.com/39789
[4 Oct 2008 14:54] Jon Stephens
Documented in the 6.0.7 changelog as follows:

        When recovering from a serial log containing many CREATE TABLESPACE and
        DROP TABLESPACE statements, Falcon could lose data from tablespaces
        not referenced by these statements.