Bug #41545 sporadic pushbuild failure: [Falcon] Error: read error on database header
Submitted: 17 Dec 2008 14:44 Modified: 15 May 14:52
Reporter: Sven Sandberg
Status: Closed
Category:Server: Falcon Severity:S2 (Serious)
Version:6.0 OS:Any
Assigned to: Vladislav Vaintroub Target Version:6.0.10
Tags: test failure, crash, sporadic, pushbuild, F_TEST
Triage: Triaged: D1 (Critical)

[17 Dec 2008 14:44] Sven Sandberg 
Description:
This failure happens in random tests. So far it happened in the falcon suite and the main
suite. The error message is:

main.have_geometry_basic                 [ fail ]

CURRENT_TEST: main.have_geometry_basic
[Falcon] Error: read error on database header
mysqltest got signal 6
read_command_buf at 0x105a900 = 
Attempting backtrace...
stack_bottom = (nil) thread_stack 0x40000
/data0/pushbuild/pb/bzr_mysql-6.0_with_tests_from_6.0-rpl/4/mysql-6.0.9-alpha-pb4/libmysqld/examples/mysqltest_embedded(my_print_stacktrace+0x2e)[0x5e88ae]
/lib64/libc.so.6[0x2b47b8c22b20]
/lib64/libc.so.6(gsignal+0x35)[0x2b47b8c22aa5]
/data0/pushbuild/pb/bzr_mysql-6.0_with_tests_from_6.0-rpl/4/mysql-6.0.9-alpha-pb4/libmysqld/examples/mysqltest_embedded(_ZN5Error5errorEPKcz+0x102)[0xa06152]
/data0/pushbuild/pb/bzr_mysql-6.0_with_tests_from_6.0-rpl/4/mysql-6.0.9-alpha-pb4/libmysqld/examples/mysqltest_embedded(_ZN2IO10readHeaderEP3Hdr+0x192)[0xa1b0a2]
/data0/pushbuild/pb/bzr_mysql-6.0_with_tests_from_6.0-rpl/4/mysql-6.0.9-alpha-pb4/libmysqld/examples/mysqltest_embedded(_ZN3Dbb4openEPKcxj+0x60)[0x9fdd50]
/data0/pushbuild/pb/bzr_mysql-6.0_with_tests_from_6.0-rpl/4/mysql-6.0.9-alpha-pb4/libmysqld/examples/mysqltest_embedded(_ZN8Database12openDatabaseEPKc+0x28)[0x9f8c28]
/data0/pushbuild/pb/bzr_mysql-6.0_with_tests_from_6.0-rpl/4/mysql-6.0.9-alpha-pb4/libmysqld/examples/mysqltest_embedded(_ZN10Connection11getDatabaseEPKcS1_P7Threads+0xcc)[0x9ed49c]
/data0/pushbuild/pb/bzr_mysql-6.0_with_tests_from_6.0-rpl/4/mysql-6.0.9-alpha-pb4/libmysqld/examples/mysqltest_embedded(_ZN10Connection12openDatabaseEPKcS1_S1_S1_S1_P7Threads+0xa7)[0x9efe07]
/data0/pushbuild/pb/bzr_mysql-6.0_with_tests_from_6.0-rpl/4/mysql-6.0.9-alpha-pb4/libmysqld/examples/mysqltest_embedded(_ZN15StorageDatabase17getOpenConnectionEv+0x58)[0x91b158]
/data0/pushbuild/pb/bzr_mysql-6.0_with_tests_from_6.0-rpl/4/mysql-6.0.9-alpha-pb4/libmysqld/examples/mysqltest_embedded(_ZN14StorageHandler10initializeEv+0x99)[0x91e269]
/data0/pushbuild/pb/bzr_mysql-6.0_with_tests_from_6.0-rpl/4/mysql-6.0.9-alpha-pb4/libmysqld/examples/mysqltest_embedded(_ZN16StorageInterface11falcon_initEPv+0x17e)[0x915e5e]
/data0/pushbuild/pb/bzr_mysql-6.0_with_tests_from_6.0-rpl/4/mysql-6.0.9-alpha-pb4/libmysqld/examples/mysqltest_embedded(_Z24ha_initialize_handlertonP13st_plugin_int+0x38)[0x81d1f8]
/data0/pushbuild/pb/bzr_mysql-6.0_with_tests_from_6.0-rpl/4/mysql-6.0.9-alpha-pb4/libmysqld/examples/mysqltest_embedded[0x65dd3a]
/data0/pushbuild/pb/bzr_mysql-6.0_with_tests_from_6.0-rpl/4/mysql-6.0.9-alpha-pb4/libmysqld/examples/mysqltest_embedded(_Z11plugin_initPiPPci+0x8ed)[0x6608bd]
/data0/pushbuild/pb/bzr_mysql-6.0_with_tests_from_6.0-rpl/4/mysql-6.0.9-alpha-pb4/libmysqld/examples/mysqltest_embedded(init_embedded_server+0x3a2)[0x5c5602]
/data0/pushbuild/pb/bzr_mysql-6.0_with_tests_from_6.0-rpl/4/mysql-6.0.9-alpha-pb4/libmysqld/examples/mysqltest_embedded(main+0x3ee)[0x5b41ee]
/lib64/libc.so.6(__libc_start_main+0xf4)[0x2b47b8c10154]
/data0/pushbuild/pb/bzr_mysql-6.0_with_tests_from_6.0-rpl/4/mysql-6.0.9-alpha-pb4/libmysqld/examples/mysqltest_embedded(__gxx_personality_v0+0xc1)[0x5a7579]
mysqltest: At line 25: Failed to flush
'/dev/shm/var-embedded-112/log/have_geometry_basic.log', errno: 28
Failed to read from '/dev/shm/var-embedded-112/log/have_geometry_basic.log', errno: 22

 - saving '/dev/shm/var-embedded-112/log/main.have_geometry_basic/' to
'/dev/shm/var-embedded-112/log/main.have_geometry_basic/'

Retrying test, attempt(2/3)...

How to repeat:
https://intranet.mysql.com/secure/pushbuild/showpush.pl?dir=bzr_mysql-6.0_with_tests_from_...
sapsrv2/embedded
xref: http://tinyurl.com/6rsuhj
[17 Dec 2008 17:21] Vladislav Vaintroub 
The actual error seems to be no space on the box. But Falcon could handle errors like that
better, without asserts
[17 Dec 2008 17:26] Bugs System 
A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/61884

2942 Vladislav Vaintroub	2008-12-17
      Bug #41545 sporadic pushbuild failure: [Falcon] Error: read error on database
header
      
      Falcon crashes in ASSERT if it cannot read header from tablespace. The actual
problem
      seems to be the lack of diskspace on the pushbuild box.
      
      Solution: throw exception instead of ASSERT. Exceptions are caught within plugin
initialization code, i.e errors like this are handled gracefully 
      with error message instead of just aborting the server.
[17 Dec 2008 17:59] Vladislav Vaintroub 
pushed to mysql-6.0-falcon-team
[13 Feb 8:23] Bugs System 
Pushed into 6.0.10-alpha (revid:alik@sun.com-20090211182317-uagkyj01fk30p1f8) (version
source revid:vvaintroub@mysql.com-20081217162604-s5f9m3idab82iuav) (merge vers:
6.0.9-alpha) (pib:6)
[15 May 14:52] MC Brown 
A note has been added to the 6.0.10 changelog: 

When the Falcon storage engine encountered an I/O error, mysqld would crash. Errors now
raise an exception, which is reported to the error log and Falcon will fail to
initialize.