MySQL Bugs: #39138: Falcon total database deletion after CREATE + DROP TABLESPACE + recovery

Bug #39138	Falcon total database deletion after CREATE + DROP TABLESPACE + recovery
Submitted:	30 Aug 2008 20:18	Modified:	4 Oct 2008 14:54
Reporter:	Philip Stoev	Email Updates:
Status:	Closed	Impact on me:	None
Category:	MySQL Server: Falcon storage engine	Severity:	S1 (Critical)
Version:	6.0-falcon-team	OS:	Any
Assigned to:	Vladislav Vaintroub	CPU Architecture:	Any

Description:
When Falcon recovers from a serial log file containing a lot of CREATE TABLESPACE and DROP TABLESPACE, the following situation occurs:

New table space FALCON_USER, id 1, type 0, filename falcon_user.fts
New table space FALCON_TEMPORARY, id 2, type 0, filename falcon_temporary.fts
Recovering database /build/bzr/6.0-falcon-team/mysql-test/var/master-data/falcon_master.fts ...
first recovery block is 1
last recovery block is 1309
recovery read block is 45
Exception: can't open file "f3": No such file or directory (2)
Couldn't open table space file "f3" for tablespace "c": can't open file "f3": No such file or directory (2)
Exception: can't open file "f3": No such file or directory (2)
IO::createFile: creating file "falcon_master.fts"

At this point the original falcon_master.fts has been obliterated. All data from all tablespaces has been lost.

How to repeat:
A test case will be provided shortly.

Suggested fix:
This is wrong on so many levels:

* IO::createFile should never ever create a file over a file that already exists, no matter what.

* The Recovery should take into account that some tablespaces may have been both created and dropped in the duration of the serial log. It should not try to open tablespace files that only existed temporarily and were later dropped. To achieve that, I assume that one of the initial recovery passes over the serial log should keep track which tablespace files remain valid throughout and which were dropped.

* An Exception during Falcon recovery should cause the server to stop. Instead, the server continues and starts accepting connections as if nothing has happened.

* Falcon files should only be created in groups of three (the falcon_master tablespace and the two serial log files) and under no circumstances should just one of those files be created while leaving the rest in an unknown state. Any time the files are not present together should result in a hard failure and server termination. In this particular bug, the serial log files were left intact while the tablespace file was recreated. This is a No, No, No (Amy Winehouse style).

Grammar file for bug 39138

Attachment: bug39138.yy (application/octet-stream, text), 581 bytes.

To reproduce this bug, please clone the latest revision of mysql-test-extra-6.0 and execute:

$ cd mysql-test/gentest
$ perl runall.pll \
  --basedir=/path/to/6.0-falcon-team \
  --grammar=conf/bug39138.yy \
  --reporters=Recovery \
  --threads=1 \
  --queries=1000

This will proceed to execute 1000 CREATE/DROP TABLESPACE/TABLE queries. After that, it will kill the server and initiate recovery. The output will show Falcon exceptions and the tablespace will be recreated. If you start the server again, you will see that SHOW TABLES continues to list varous tables, but they are all unreadable, regardless of what tablespace they were defined in.

A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

http://lists.mysql.com/commits/53247

2808 Vladislav Vaintroub 2008-09-04
Bug#39138 : Falcon total database deletion after CREATE + DROP TABLESPACE + recovery

Problem: Falcon misinterpretins some errors from openDatabase(), assumes database does
not exist and tries to create a new database. During creation, it overwrites already existing
files (by passing O_CREAT|O_TRUNC to open() on Posix, and CREATE_ALWAYS to CreateFile on Windows).

This is corrected in this patch

- open() is now called with O_CREAT|O_EXCL and CreateFile() with CREATE_NEW.
This will make Falcon never overwrite existing files.

- To ensure the proper cleanup of files created during create database attempt,
"create" status is tracked. On error, createDatabase sets global deleteFilesOnExit
flag - this causes new files to be deleted in destructors of IO and SerialLogFile.
But there is no cleanup if process terminates with exception/signal.
A more elegant solution for cleanup is possible on file systems that support user-mode
transactions (like recent NTFS), by making the whole createDatabase/openDatabase a single
transaction with commit() at the end. Unfortunately this solution is not portable yet,
not even on downlevel Windows ,so it is not implemented.

- We also ensure with ASSERT, that createDatabase never opens existing files, but always
creates new ones.
- We do not attempt to create a database after failing recovery anymore.

I have reviewed this patch and it has been pushed. Philip, can you verify that this fix is complete?

The biggest recovery issue mentioned in this bug -- the complete deletion of all tablespaces, has been resolved by this patch. Upon Vlad's request, a separate bug will be filed to track the rest of the recovery issues involving tablespaces.

The new bug is 

http://bugs.mysql.com/39789

Documented in the 6.0.7 changelog as follows:

        When recovering from a serial log containing many CREATE TABLESPACE and
        DROP TABLESPACE statements, Falcon could lose data from tablespaces
        not referenced by these statements.