Bug #41545 sporadic pushbuild failure: [Falcon] Error: read error on database header
Submitted: 17 Dec 2008 13:44 Modified: 15 May 2009 12:52
Reporter: Sven Sandberg Email Updates:
Status: Closed Impact on me:
None 
Category:MySQL Server: Falcon storage engine Severity:S2 (Serious)
Version:6.0 OS:Any
Assigned to: Vladislav Vaintroub CPU Architecture:Any
Tags: crash, F_TEST, pushbuild, sporadic, test failure
Triage: Triaged: D1 (Critical)

[17 Dec 2008 13:44] Sven Sandberg
Description:
This failure happens in random tests. So far it happened in the falcon suite and the main suite. The error message is:

main.have_geometry_basic                 [ fail ]

CURRENT_TEST: main.have_geometry_basic
[Falcon] Error: read error on database header
mysqltest got signal 6
read_command_buf at 0x105a900 = 
Attempting backtrace...
stack_bottom = (nil) thread_stack 0x40000
/data0/pushbuild/pb/bzr_mysql-6.0_with_tests_from_6.0-rpl/4/mysql-6.0.9-alpha-pb4/libmysqld/examples/mysqltest_embedded(my_print_stacktrace+0x2e)[0x5e88ae]
/lib64/libc.so.6[0x2b47b8c22b20]
/lib64/libc.so.6(gsignal+0x35)[0x2b47b8c22aa5]
/data0/pushbuild/pb/bzr_mysql-6.0_with_tests_from_6.0-rpl/4/mysql-6.0.9-alpha-pb4/libmysqld/examples/mysqltest_embedded(_ZN5Error5errorEPKcz+0x102)[0xa06152]
/data0/pushbuild/pb/bzr_mysql-6.0_with_tests_from_6.0-rpl/4/mysql-6.0.9-alpha-pb4/libmysqld/examples/mysqltest_embedded(_ZN2IO10readHeaderEP3Hdr+0x192)[0xa1b0a2]
/data0/pushbuild/pb/bzr_mysql-6.0_with_tests_from_6.0-rpl/4/mysql-6.0.9-alpha-pb4/libmysqld/examples/mysqltest_embedded(_ZN3Dbb4openEPKcxj+0x60)[0x9fdd50]
/data0/pushbuild/pb/bzr_mysql-6.0_with_tests_from_6.0-rpl/4/mysql-6.0.9-alpha-pb4/libmysqld/examples/mysqltest_embedded(_ZN8Database12openDatabaseEPKc+0x28)[0x9f8c28]
/data0/pushbuild/pb/bzr_mysql-6.0_with_tests_from_6.0-rpl/4/mysql-6.0.9-alpha-pb4/libmysqld/examples/mysqltest_embedded(_ZN10Connection11getDatabaseEPKcS1_P7Threads+0xcc)[0x9ed49c]
/data0/pushbuild/pb/bzr_mysql-6.0_with_tests_from_6.0-rpl/4/mysql-6.0.9-alpha-pb4/libmysqld/examples/mysqltest_embedded(_ZN10Connection12openDatabaseEPKcS1_S1_S1_S1_P7Threads+0xa7)[0x9efe07]
/data0/pushbuild/pb/bzr_mysql-6.0_with_tests_from_6.0-rpl/4/mysql-6.0.9-alpha-pb4/libmysqld/examples/mysqltest_embedded(_ZN15StorageDatabase17getOpenConnectionEv+0x58)[0x91b158]
/data0/pushbuild/pb/bzr_mysql-6.0_with_tests_from_6.0-rpl/4/mysql-6.0.9-alpha-pb4/libmysqld/examples/mysqltest_embedded(_ZN14StorageHandler10initializeEv+0x99)[0x91e269]
/data0/pushbuild/pb/bzr_mysql-6.0_with_tests_from_6.0-rpl/4/mysql-6.0.9-alpha-pb4/libmysqld/examples/mysqltest_embedded(_ZN16StorageInterface11falcon_initEPv+0x17e)[0x915e5e]
/data0/pushbuild/pb/bzr_mysql-6.0_with_tests_from_6.0-rpl/4/mysql-6.0.9-alpha-pb4/libmysqld/examples/mysqltest_embedded(_Z24ha_initialize_handlertonP13st_plugin_int+0x38)[0x81d1f8]
/data0/pushbuild/pb/bzr_mysql-6.0_with_tests_from_6.0-rpl/4/mysql-6.0.9-alpha-pb4/libmysqld/examples/mysqltest_embedded[0x65dd3a]
/data0/pushbuild/pb/bzr_mysql-6.0_with_tests_from_6.0-rpl/4/mysql-6.0.9-alpha-pb4/libmysqld/examples/mysqltest_embedded(_Z11plugin_initPiPPci+0x8ed)[0x6608bd]
/data0/pushbuild/pb/bzr_mysql-6.0_with_tests_from_6.0-rpl/4/mysql-6.0.9-alpha-pb4/libmysqld/examples/mysqltest_embedded(init_embedded_server+0x3a2)[0x5c5602]
/data0/pushbuild/pb/bzr_mysql-6.0_with_tests_from_6.0-rpl/4/mysql-6.0.9-alpha-pb4/libmysqld/examples/mysqltest_embedded(main+0x3ee)[0x5b41ee]
/lib64/libc.so.6(__libc_start_main+0xf4)[0x2b47b8c10154]
/data0/pushbuild/pb/bzr_mysql-6.0_with_tests_from_6.0-rpl/4/mysql-6.0.9-alpha-pb4/libmysqld/examples/mysqltest_embedded(__gxx_personality_v0+0xc1)[0x5a7579]
mysqltest: At line 25: Failed to flush '/dev/shm/var-embedded-112/log/have_geometry_basic.log', errno: 28
Failed to read from '/dev/shm/var-embedded-112/log/have_geometry_basic.log', errno: 22

 - saving '/dev/shm/var-embedded-112/log/main.have_geometry_basic/' to '/dev/shm/var-embedded-112/log/main.have_geometry_basic/'

Retrying test, attempt(2/3)...

How to repeat:
https://intranet.mysql.com/secure/pushbuild/showpush.pl?dir=bzr_mysql-6.0_with_tests_from_... sapsrv2/embedded
xref: http://tinyurl.com/6rsuhj
[17 Dec 2008 16:21] Vladislav Vaintroub
The actual error seems to be no space on the box. But Falcon could handle errors like that better, without asserts
[17 Dec 2008 16:26] Bugs System
A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/61884

2942 Vladislav Vaintroub	2008-12-17
      Bug #41545 sporadic pushbuild failure: [Falcon] Error: read error on database header
      
      Falcon crashes in ASSERT if it cannot read header from tablespace. The actual problem
      seems to be the lack of diskspace on the pushbuild box.
      
      Solution: throw exception instead of ASSERT. Exceptions are caught within plugin initialization code, i.e errors like this are handled gracefully 
      with error message instead of just aborting the server.
[17 Dec 2008 16:59] Vladislav Vaintroub
pushed to mysql-6.0-falcon-team
[13 Feb 2009 7:23] Bugs System
Pushed into 6.0.10-alpha (revid:alik@sun.com-20090211182317-uagkyj01fk30p1f8) (version source revid:vvaintroub@mysql.com-20081217162604-s5f9m3idab82iuav) (merge vers: 6.0.9-alpha) (pib:6)
[15 May 2009 12:52] MC Brown
A note has been added to the 6.0.10 changelog: 

When the Falcon storage engine encountered an I/O error, mysqld would crash. Errors now raise an exception, which is reported to the error log and Falcon will fail to initialize.