MySQL Bugs: #32301: Falcon storage engine does not support consistent read via handler methods

Bug #32301	Falcon storage engine does not support consistent read via handler methods
Submitted:	13 Nov 2007 0:57	Modified:	3 Dec 2007 14:26
Reporter:	Chuck Bell	Email Updates:
Status:	Closed
Category:	Server: Falcon	Severity:	S3 (Non-critical)
Version:	6.0	OS:	Any
Assigned to:	Kevin Lewis	Target Version:

Description:
I have been working with the mysql-6.0 code (from the tree of the same name) and noticed
that the falcon engine doesn't support consistent snapshot via the handler methods -- at
least not in the manner expected. 

Does falcon support consistent read? If it does, then we need some additional
functionality added to falcon in order to read falcon tables in online backup via
consistent read.

Allow me to explain how we are using consistent read with InnoDB.

When a table is included in a backup, the online backup code interrogates the handler for
the table to see if the start_consistent_snapshot() method is implemented. If it is, then
that indicates (to us) that the engine supports consistent read and we assign that table
to the consistent snapshot driver. If the storage engine does not have that method
defined, the table is assigned to the default (blocking) driver.

Once we identify a table's storage engine as supporting consistent read and assign it to
the consistent snapshot driver, we then use the following code to initiate the consistent
read in our consistent snapshot driver's lock() method.

  m_thd->lex->start_transaction_opt|= MYSQL_START_TRANS_OPT_WITH_CONS_SNAPSHOT;
  int res= begin_trans(m_thd);

The table is read using normal handler methods for a table scan, e.g. hdl->ha_rnd_init(1)
+ hton->rnd_next() + hdl->ha_rnd_end().

If falcon supports consistent read and therefore will use the consistent snapshot driver
to do a logical backup of tables, then the falcon engine must support these methods and
capabilities:

  hton->start_consistent_snapshot();
  m_thd->lex->start_transaction_opt|= MYSQL_START_TRANS_OPT_WITH_CONS_SNAPSHOT;
  ha_start_consistent_snapshot();
  ..and, of course, transactions. 

See ha_innodb.cc for details on how InnoDB implements the above.

How to repeat:
Create a table using the falcon storage engine.
Attempt a backup while inserting data into the table.

Since the falcon engine doesn't use the handler methods, online backup will see the
storage engine as not supporting consistent read and will use the default (blocking)
driver. Thus, inserts that occur at the same time as backup (provided there are no lock
issues) are included in the backup image.

Suggested fix:
Change the falcon storage engine to support the handler methods as stated above.

I have compiled Kevin's code in mysql-6.0-falcon-team after pulling from mysql-6.0-backup
and applying the patch for BUG#31383. 

The code compiles well on Windows (great job, Falcon team!) and online backup correctly
identifies the falcon table as supporting consistent read. However, when the consistent
snapshot test was run, the backup image included rows inserted during the period when the
consistent read should have been activated.

I have included a modified version of the backup_snapshot test that can be used to test
whether the falcon storage engine is correctly engaging the consistent read. Please see
WL#4120 for details on how the test was constructed and how to set breakpoints in the
backup code to ensure deterministic results.

NOTICE: This bug report, once solved, will require BUG#31383 and the code from
mysql-6.0-backup (if BUG#31383 has not been included already).

Modified consistent snapshot test (includes falcon tables).

Attachment: backup_snapshot.test (application/octet-stream, text), 5.35 KiB.

In ha_falcon.cpp, StorageInterface::start_consistent_snapshot(),  I used the wrong
transaction isolation level;

int ret = storageHandler->startTransaction(thd, TRANSACTION_READ_COMMITTED);

It should be 

int ret = storageHandler->startTransaction(thd, TRANSACTION_CONSISTENT_READ);

I debugged it with Chuck Bell's merged tree and test case, and it does start the
transaction OK.  I pushed this into the mysql-6.0-falcon-team tree.

Checked in another change which allowed the test case to work.  Forwarded a patch to Chuck
for verification.

I can confirm that the changes descibed above fix the problem.

In addition, I think it best to add falcon to the existing backup_snapshot test. I have
attached a patch to do this. A corresponding result file can be generated. 

Note: The new test should be pushed only after the mysql-6.0-backup tree is merged with
the mysql-6.0-falcon-team tree (via the normal process). This is because the corrected
test requires the latest patches from mysql-6.0-backup and mysql-6.0-falcon-team.
Specifically, the patch for BUG#31383 (which was pushed on 15 Nov) and the falcon code
updates.

Diffs to add falcon table to backup_snapshot.test.

Attachment: falcon_snapshot.diffs (application/octet-stream, text), 6.24 KiB.

Sample result file for updated backup_snapshot test.

Attachment: backup_snapshot.result (application/octet-stream, text), 5.69 KiB.

Upgraded because of need to provide support for Online Backup Consistent Snapshot backup
in Falcon in 6.0.4

Chuck,  Is there a testcase or test sequence that shows what is still wrong with this
consistent snapshot code?

> From: Kevin Lewis [mailto:klewis@mysql.com] 
> Sent: Thursday, November 29, 2007 6:17 AM
> I do not see any problems with Falcon consistent snapshot.  I 
> ran Chuck's latest testcase and code with a break on the 
> backup wait, and was sure that during the backup, another 
> client inserted and updated the table.  After the restore, 
> the new records were not visible.

-----Original Message-----
From: Chuck Bell [mailto:cbell@mysql.com] 
Sent: Thursday, November 29, 2007 8:57 AM

I concur. The falcon code works. I did a rebuild and the issue I was seeing
went away. Operator error let's call it. ;)

Pushed into 6.0.4-alpha

A note has been added to the 6.0.4 changelog: 

Falcon would incorrectly return the supported repeatable-read level when queried by the
Online Backup system, preventing the ability to create a consistent snapshot backup.