Description:
Recovery fails following a forced server crash (kill -9) after doing BLOB operations using the Falcon engine.
Example error (from running Pushbuild 2 test "falcon_blob_recovery"):
# 13:01:49 Sending SIGKILL to mysqld with pid 23821 in order to force a recovery.
# 13:02:01 Attempting database recovery...
# 13:02:01 Executing /export/home/pb2/test/sb_1-208875-1230552017.14/mysql-6.0.10-alpha-linux-i686-test/libexec/mysqld --no-defaults --core-file --loose-console --loose-falcon-debug-mask=65535 --language=/export/home/pb2/test/sb_1-208875-1230552017.14/mysql-6.0.10-alpha-linux-i686-test/share/mysql/english/ --datadir="/export/home/pb2/test/sb_1-208875-1230552017.14/mysql-6.0.10-alpha-linux-i686-test/mysql-test/var/master-data_recovery" --socket="/export/home/pb2/test/sb_1-208875-1230552017.14/tmp/b2kG26PMP7/master.sock" --port=19306 2>&1 .
# 13:02:01 New table space FALCON_USER, id 1, type 0, filename falcon_user.fts
# 13:02:01 Recovering from lost page inventory page 4
# 13:02:01 New table space FALCON_TEMPORARY, id 2, type 0, filename falcon_temporary.fts
# 13:02:01 Recovering database /dev/shm/var/master-data_recovery/falcon_master.fts ...
# 13:02:01 first recovery block is 1
# 13:02:01 last recovery block is 165
# 13:02:01 recovery read block is 65
# 13:02:11 [Falcon] Error: page 98015/1 wrong page type, expected 8 got 0
# 13:02:11
# 13:02:11 Bugcheck: page 98015/1 wrong page type, expected 8 got 0
# 13:02:11
# 13:02:11 081229 13:02:11 - mysqld got signal 6 ;
# 13:02:11 This could be because you hit a bug. It is also possible that this binary
# 13:02:11 or one of the libraries it was linked against is corrupt, improperly built,
# 13:02:11 or misconfigured. This error can also be caused by malfunctioning hardware.
# 13:02:11 We will try our best to scrape up some info that will hopefully help diagnose
# 13:02:11 the problem, but since we have already crashed, something is definitely wrong
# 13:02:11 and this may fail.
# 13:02:11
# 13:02:11 key_buffer_size=8384512
# 13:02:11 read_buffer_size=131072
# 13:02:11 max_used_connections=0
# 13:02:11 max_threads=151
# 13:02:11 thread_count=0
# 13:02:11 connection_count=0
# 13:02:11 It is possible that mysqld could use up to
# 13:02:11 key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 337813 K
# 13:02:11 bytes of memory
# 13:02:11 Hope that's ok; if not, decrease some variables in the equation.
# 13:02:11
# 13:02:11 thd: 0x0
# 13:02:11 Attempting backtrace. You can use the following information to find out
# 13:02:11 where mysqld died. If you see no messages after this, something went
# 13:02:11 terribly wrong...
# 13:02:11 stack_bottom = (nil) thread_stack 0x30c00
# 13:02:11 /export/home/pb2/test/sb_1-208875-1230552017.14/mysql-6.0.10-alpha-linux-i686-test/libexec/mysqld(my_print_stacktrace+0x32) [0x880c335]
# 13:02:11 /export/home/pb2/test/sb_1-208875-1230552017.14/mysql-6.0.10-alpha-linux-i686-test/libexec/mysqld(handle_segfault+0x2cd) [0x82b8af7]
# 13:02:11 [0xf13420]
# 13:02:11 /export/home/pb2/test/sb_1-208875-1230552017.14/mysql-6.0.10-alpha-linux-i686-test/libexec/mysqld(Error::debugBreak()+0x12) [0x8547988]
# 13:02:11 /export/home/pb2/test/sb_1-208875-1230552017.14/mysql-6.0.10-alpha-linux-i686-test/libexec/mysqld(Error::error(char const*, ...)+0x7a) [0x8547a04]
# 13:02:11 /export/home/pb2/test/sb_1-208875-1230552017.14/mysql-6.0.10-alpha-linux-i686-test/libexec/mysqld(Cache::fetchPage(Dbb*, int, PageType, LockType)+0x438) [0x8638884]
# 13:02:11 /export/home/pb2/test/sb_1-208875-1230552017.14/mysql-6.0.10-alpha-linux-i686-test/libexec/mysqld(Dbb::fetchPage(int, PageType, LockType)+0x48) [0x85a089e]
# 13:02:11 /export/home/pb2/test/sb_1-208875-1230552017.14/mysql-6.0.10-alpha-linux-i686-test/libexec/mysqld(PageInventoryPage::isPageInUse(Dbb*, int)+0x7b) [0x85d86fb]
# 13:02:11 /export/home/pb2/test/sb_1-208875-1230552017.14/mysql-6.0.10-alpha-linux-i686-test/libexec/mysqld(Cache::fetchPage(Dbb*, int, PageType, LockType)+0xe4) [0x8638530]
# 13:02:11 /export/home/pb2/test/sb_1-208875-1230552017.14/mysql-6.0.10-alpha-linux-i686-test/libexec/mysqld(Dbb::fetchPage(int, PageType, LockType)+0x48) [0x85a089e]
# 13:02:11 /export/home/pb2/test/sb_1-208875-1230552017.14/mysql-6.0.10-alpha-linux-i686-test/libexec/mysqld(Section::redoDataPage(int, int)+0x91) [0x8603a2b]
# 13:02:11 /export/home/pb2/test/sb_1-208875-1230552017.14/mysql-6.0.10-alpha-linux-i686-test/libexec/mysqld(Dbb::redoDataPage(int, int, int)+0x34) [0x85a1a26]
# 13:02:11 /export/home/pb2/test/sb_1-208875-1230552017.14/mysql-6.0.10-alpha-linux-i686-test/libexec/mysqld(SRLDataPage::pass2()+0x12d) [0x864d1bd]
# 13:02:11 /export/home/pb2/test/sb_1-208875-1230552017.14/mysql-6.0.10-alpha-linux-i686-test/libexec/mysqld(SerialLog::recover()+0x915) [0x860e94d]
# 13:02:11 /export/home/pb2/test/sb_1-208875-1230552017.14/mysql-6.0.10-alpha-linux-i686-test/libexec/mysqld(Database::openDatabase(char const*)+0x376) [0x85992d6]
# 13:02:11 /export/home/pb2/test/sb_1-208875-1230552017.14/mysql-6.0.10-alpha-linux-i686-test/libexec/mysqld(Connection::getDatabase(char const*, char const*, Threads*)+0x149) [0x858cd13]
# 13:02:11 /export/home/pb2/test/sb_1-208875-1230552017.14/mysql-6.0.10-alpha-linux-i686-test/libexec/mysqld(Connection::openDatabase(char const*, char const*, char const*, char const*, char const*, Threads*)+0x198) [0x858e22e]
# 13:02:11 /export/home/pb2/test/sb_1-208875-1230552017.14/mysql-6.0.10-alpha-linux-i686-test/libexec/mysqld(StorageDatabase::getOpenConnection()+0x86) [0x8551c2a]
# 13:02:11 /export/home/pb2/test/sb_1-208875-1230552017.14/mysql-6.0.10-alpha-linux-i686-test/libexec/mysqld(StorageHandler::initialize()+0x9f) [0x85542d5]
# 13:02:11 /export/home/pb2/test/sb_1-208875-1230552017.14/mysql-6.0.10-alpha-linux-i686-test/libexec/mysqld(StorageInterface::falcon_init(void*)+0x23f) [0x8545433]
# 13:02:11 /export/home/pb2/test/sb_1-208875-1230552017.14/mysql-6.0.10-alpha-linux-i686-test/libexec/mysqld(ha_initialize_handlerton(st_plugin_int*)+0x99) [0x83f8671]
# 13:02:11 /export/home/pb2/test/sb_1-208875-1230552017.14/mysql-6.0.10-alpha-linux-i686-test/libexec/mysqld [0x84abfb0]
# 13:02:11 /export/home/pb2/test/sb_1-208875-1230552017.14/mysql-6.0.10-alpha-linux-i686-test/libexec/mysqld(plugin_init(int*, char**, int)+0x5f4) [0x84ad3b2]
# 13:02:11 /export/home/pb2/test/sb_1-208875-1230552017.14/mysql-6.0.10-alpha-linux-i686-test/libexec/mysqld [0x82bb6ff]
# 13:02:11 /export/home/pb2/test/sb_1-208875-1230552017.14/mysql-6.0.10-alpha-linux-i686-test/libexec/mysqld(main+0x1d9) [0x82bc6a5]
# 13:02:11 /lib/libc.so.6(__libc_start_main+0xdc) [0x736dec]
# 13:02:11 /export/home/pb2/test/sb_1-208875-1230552017.14/mysql-6.0.10-alpha-linux-i686-test/libexec/mysqld(__gxx_personality_v0+0x3cd) [0x81d3bb1]
How to repeat:
Branch the mysql-test-extra-6.0 branch and the mysql-6.0-falcon-team branch (referred to below as $srcdir). In the test-extra branch, cd to mysql-test/gentest.
On a linux host, run the following command in order to run the falcon_blob_recovery test:
./runall.pl \
--basedir=$srcdir \
--mysqld=--loose-innodb-lock-wait-timeout=1 \
--mysqld=--table-lock-wait-timeout=1 \
--mysqld=--loose-falcon-lock-wait-timeout=1 \
--mysqld=--loose-falcon-debug-mask=2 \
--mysqld=--skip-safemalloc \
--grammar=conf/falcon_blobs.yy \
--gendata=conf/falcon_blobs.zz \
--duration=130 \
--mem \
--threads=1 \
--reporters=Deadlock,ErrorLog,Backtrace,Recovery,WinPackage \
--mysqld=--log-output=file \
--queries=100000 \
--engine=falcon
Suggested fix:
Make Falcon recovery work with BLOBs.
