MySQL Bugs: #44114: Falcon recovery Error: page 5342/1 wrong page type, expected 5 got 511

Bug #44114	Falcon recovery Error: page 5342/1 wrong page type, expected 5 got 511
Submitted:	6 Apr 2009 13:15	Modified:	15 May 2009 15:52
Reporter:	Philip Stoev	Email Updates:
Status:	Closed	Impact on me:	None
Category:	MySQL Server: Falcon storage engine	Severity:	S1 (Critical)
Version:	6.0-falcon-team	OS:	Any
Assigned to:	Vladislav Vaintroub	CPU Architecture:	Any

Description:
Since all bugs for "wrong page type, expected 5" are now closed, and this error continues to happen, I am filing a new bug.

# 12:12:07 [Falcon] Error: page 5342/1 wrong page type, expected 5 got 511

mysqld(Error::debugBreak()+0xe) [0x9f1b24]
mysqld(Error::error(char const*, ...)+0x131) [0x9f1c57]
mysqld(Cache::fetchPage(Dbb*, int, PageType, LockType)+0x4cc) [0x9c0298]
mysqld(Dbb::fetchPage(int, PageType, LockType)+0x4c) [0x9e70d4]
mysqld(Dbb::handoffPage(Bdb*, int, PageType, LockType)+0x2c) [0x9e72fc]
mysqld(IndexRootPage::findInsertionLeaf(Dbb*, int, IndexKey*, int, unsigned int, bool*)+0x246) [0xa02$
mysqld(IndexRootPage::indexMerge(Dbb*, int, SRLUpdateIndex*, unsigned int)+0x16f) [0xa03323]
mysqld(SRLUpdateIndex::execute()+0x9f) [0xa75d5d]
mysqld(SRLUpdateIndex::redo()+0x15) [0xa75d77]
mysqld(SerialLog::recover()+0xd06) [0xa56f74]
mysqld(Database::openDatabase(char const*)+0x407) [0x9e068b]
mysqld(Connection::getDatabase(char const*, char const*, Threads*)+0x16a) [0x9cec84]
mysqld(Connection::openDatabase(char const*, char const*, char const*, char const*, char const*, Thre$
mysqld(StorageDatabase::getOpenConnection()+0x89) [0x985b7d]
mysqld(StorageHandler::initialize()+0xa3) [0x988c0f]
mysqld(StorageInterface::falcon_init(void*)+0x2ac) [0x97ef94]
mysqld(ha_initialize_handlerton(st_plugin_int*)+0xae) [0x820ea2]
mysqld [0x8d89e0]
mysqld(plugin_init(int*, char**, int)+0x673) [0x8dc1d9]
mysqld [0x6ca7f7]
mysqld(main+0x1f0) [0x6cb2f5]

How to repeat:
The tablespace will be uploaded shortly. It was produced with the following command:

$ perl runall.pl    --engine=Falcon   --reporters=Deadlock,ErrorLog,Backtrace,Recovery,Shutdown   --mysqld=--loose-falcon-lock-wait-timeout=1   --mysqld=--loose-innodb-lock-wait-timeout=1   --mysqld=--log-output=none   --mysqld=--skip-safemalloc   --mysqld=--transaction-isolation=READ-COMMITTED --mysqld=--falcon-page-size=32K  --rows=10000 --threads=4 --mysqld=--falcon-use-deferred-index-hash=1   --mask=57022   --queries=100000000   --duration=900   --basedir=/build/bzr/6.0-falcon-team  --gendata=conf/combinations.zz  --grammar=conf/combinations.yy  --vardir=/ide/april05/simple/current  

Note, --mysqld=--falcon-use-deferred-index-hash=1 which may or may not be important.

Tablespace before recovery

http://mysql-systemqa.s3.amazonaws.com/bug44114-2.zip

This failure comes in the following 4 forms:

[Falcon] Error: page 569/1 wrong page type, expected 5 got 2596
[Falcon] Error: page 1906/1 wrong page type, expected 5 got 7
[Falcon] Error: page 1104/1 wrong page type, expected 5 got 9
[Falcon] Error: page 21/1 wrong page type, expected 5 got 511

and is very widespread in PB2 with dozens of occurrences, including with default Falcon startup options.

A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/71568

3105 Vladislav Vaintroub	2009-04-07
      Bug #44114 Falcon recovery Error: page 5342/1 wrong page type, expected 5 got 511
      
      This patch makes sure pages written are valid (at least they have valid page types). 
      Falcon will assert if garbage data is written to the tablespaces. This will make errors like "got page type 511" impossible. The actual error that
      causes page corruption  will be simpler to find.

A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/71569

3105 Vladislav Vaintroub	2009-04-07
      Bug #44114 Falcon recovery Error: page 5342/1 wrong page type, expected 5 got 511
      
      This patch makes sure pages written are valid (at least they have valid page types and page numbers) 
      Falcon will assert if garbage data is written to the tablespaces. This will make errors like "got page type 511" impossible. The actual error that
      causes page corruption  will be simpler to find.

Vlad's patch will be definitive, but I suspect that the problem is
not that we're writing bad pages, but that we sometimes read a page
number that's no longer valid by reading beyond the maxLine on a
RecordLocator or Section page.

Pushed into 6.0.11-alpha (revid:hky@sun.com-20090415164923-9arx29ye5pzxd4zf) (version source revid:vvaintroub@mysql.com-20090407182323-qohwrke048hvp8hn) (merge vers: 6.0.11-alpha) (pib:6)

An entry has been added to the 6.0.11 changelog: 

Recovery of Falcon tables could fail with an indicating that a wrong page type was identified in the Falcon serial lo