MySQL Bugs: #40636: falcon_bug_28095 test fails with "Server failed to restart"

Bug #40636	falcon_bug_28095 test fails with "Server failed to restart"
Submitted:	11 Nov 2008 13:25	Modified:	19 Dec 2008 13:17
Reporter:	John H. Embretsen
Status:	Duplicate
Category:	Tests	Severity:	S3 (Non-critical)
Version:	6.0-bzr	OS:	Linux
Assigned to:	Andrei Elkin	Target Version:	6.0-beta
Tags:	F_TEST, 6.0-rpl-green, valgrind, pushbuild, test failure, falcon
Triage:	Triaged: D1 (Critical)

Description:
The test falcon.falcon_bug_28095 fails when run with Valgrind.

Test failure symptoms are:

falcon.falcon_bug_28095        [ fail ]

../libtool: line 6823: LC_MESSAGES: command not found
==17708== Memcheck, a memory error detector.
==17708== Copyright (C) 2002-2006, and GNU GPL'd, by Julian Seward et al.
==17708== Using LibVEX rev 1658, a library for dynamic binary translation.
==17708== Copyright (C) 2004-2006, and GNU GPL'd, by OpenWorks LLP.
==17708== Using valgrind-3.2.1, a dynamic binary instrumentation framework.
==17708== Copyright (C) 2000-2006, and GNU GPL'd, by Julian Seward et al.
==17708== For more details, rerun with: -v
==17708==
mysqltest: In included file "./include/wait_until_connected_again.inc": At line 23:
Server failed to restart

The result from queries just before the failure was:
< snip >
SET @@storage_engine = 'Falcon';
DROP TABLE IF EXISTS t1;
DROP TABLE IF EXISTS t2;
CREATE TABLE t1 (a int);
INSERT INTO t1 VALUES (1);
INSERT INTO t1 VALUES (2);
INSERT INTO t1 VALUES (3);
SELECT * FROM t1 ORDER BY a;
a
1
2
3
CREATE TABLE t2 (b int);
SELECT @@GLOBAL.falcon_debug_mask;
@@GLOBAL.falcon_debug_mask
0
SET GLOBAL falcon_debug_mask = 4096;
SELECT @@GLOBAL.falcon_debug_mask;
@@GLOBAL.falcon_debug_mask
4096

More results from queries before failure can be found in
/export/home/tmp/je159969/mysql/bzr-shared/jsandbox/mysql-test/var/log/falcon_bug_28095.log
==17708==
==17708== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 24 from 1)
==17708== malloc/free: in use at exit: 28 bytes in 1 blocks.
==17708== malloc/free: 1,952 allocs, 1,951 frees, 5,603,980 bytes allocated.
==17708== For counts of detected errors, rerun with: -v
==17708== searching for pointers to 1 not-freed blocks.
==17708== checked 2,067,376 bytes.
==17708==
==17708== LEAK SUMMARY:
==17708==    definitely lost: 0 bytes in 0 blocks.
==17708==      possibly lost: 0 bytes in 0 blocks.
==17708==    still reachable: 0 bytes in 0 blocks.
==17708==         suppressed: 28 bytes in 1 blocks.

Aborting: falcon.falcon_bug_28095 failed in default mode.
To continue, re-run with '--force'.
Stopping All Servers
mysql-test-run: WARNING: Forcing kill of process 17838

How to repeat:
Example for Linux/Solaris:

./mysql-test-run.pl --valgrind --suite=falcon falcon_bug_28095

Suggested fix:
1. (temp) Disable this test for Valgrind runs.
2. Make sure "soft server restart" (mysql-test feature, see Bug#39205) works with
Valgrind.
3. When 2 is done, re-enable this test for valgrind.

I plan to disable this test for valgrind test runs in Pushbuild, so assigning this to
myself for now. 

Real fix (probably in mysql-test framework) requires more study, related to for example
Bug#39025 (--shutdown_server does not work).

See also mail to falcon@lists.mysql.com about this mysql-test feature:
http://lists.mysql.com/falcon/147

A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/58448

2907 John H. Embretsen	2008-11-11
      Disable falcon_bug_28095 in valgrind runs, as 'soft' server restart feature of
mysql-test does not seem to work in such an environment. See Bug#40636 - falcon_bug_28095
test fails with 'Server failed to restart' in valgrind runs.

Patch which disables this test in valgrind runs has just been pushed to
mysql-6.0-falcon-team branch.
Unassigning myself as I am not looking into the root cause of this failure at this time.

This test should be closely monitored for further instabilities. If it continues to fail
in other configurations (seems to have happended once on the vm-win2003-32-a platform in
Pushbuild), the entire test should be disabled until the root cause is fixed.

A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/58651

2911 Hakan Kuecuekyilmaz	2008-11-13
      Fix for Bug#40636
          falcon_bug_28095 test fails with "Server failed to restart" in valgrind runs.

Looks good, I hope this works for all tests using restart_mysqld.inc for a long time to
come :) OK to push.

Pushed into 6.0.9-alpha  (revid:hky@sun.com-20081113145542-lq0szqt4ss2gp7v0) (version
source revid:hky@sun.com-20081113145542-lq0szqt4ss2gp7v0) (pib:5)

Re-opening bug since this test still fails with the same message in 6.0-rpl and
6.0_with_test_from_6.0-rpl, after this patch was merged to the tree.

E.g., here:
https://intranet.mysql.com/secure/pushbuild/showpush.pl?dir=bzr_mysql-6.0-rpl&order=110
sapsrv1/falcon

and here, respectively:
https://intranet.mysql.com/secure/pushbuild/showpush.pl?dir=bzr_mysql-6.0_with_tests_from_...
sapsrv2/falcon (WARNING: this url links to a very big push. It can cause Firefox to
freeze. Press ESC while the page loads.)

Here is an xref link showing where it fails: http://tinyurl.com/5vxky2

Changed synopsis to reflect the fact that the test fails this way also in non-valgrind
runs.

Failure seems to be reproducible always if MTR v2 is used (*-rpl branches and
mysql-6.0_with_tests_from_6.0-rpl branch).

Test fails with two symptoms:

falcon.falcon_bug_28095        [ fail ]
mysqltest: At line 50: query 'SELECT * FROM t1 ORDER BY a' failed: 2006: MySQL server has
gone away

and

falcon.falcon_bug_28095        [ fail ]
mysqltest: In included file "./include/wait_until_connected_again.inc": At line 23:
Server failed to restart

Looks like Guilhem found a bug in wait_until_disconnected.inc

let $mysql_errno= 9999;
while (!$mysql_errno)

should be

let $mysql_errno= 0;
while (!$mysql_errno)

Must be the cause of this final problems.

Applying the change leads to
falcon.falcon_bug_28095        [ fail ]

mysqltest: In included file "./include/wait_until_disconnected.inc": At line 14: query
'show status' failed with wrong errno 2013: 'Lost connection to MySQL server during
query', instead of 0...

I added 2013 to the list of allowed errors.

Now it fails with:

falcon.falcon_bug_28095        [ fail ]

mysqltest: In included file "./include/wait_until_connected_again.inc": At line 23:
Server failed to restart

A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/61272

2705 Magnus Svensson	2008-12-10
      Bug#40636 falcon_bug_28095 test fails with "Server failed to restart"

Test still fails.

Ack, at least it's consistent now.

A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/61645

2743 He Zhenxing	2008-12-15 [merge]
      Auto merge

Taking over, as the bug is actually same as Bug #41017, which is being fixed.

Pushed into 6.0.10-alpha (revid:luis.soares@sun.com-20090129165607-wiskabxm948yx463)
(version source revid:luis.soares@sun.com-20090129163120-e2ntks4wgpqde6zt) (merge vers:
6.0.10-alpha) (pib:6)

Pushed into 5.1.32 (revid:luis.soares@sun.com-20090129165946-d6jnnfqfokuzr09y) (version
source revid:msvensson@mysql.com-20081210184027-tyfqr3fycylp9988) (merge vers: 5.1.31)
(pib:6)

Pushed into 5.1.32-ndb-6.3.23 (revid:tomas.ulin@sun.com-20090217131017-6u8qz1edkjfiobef)
(version source revid:tomas.ulin@sun.com-20090203133556-9rclp06ol19bmzs4) (merge vers:
5.1.32-ndb-6.3.22) (pib:6)

Pushed into 5.1.32-ndb-6.4.3 (revid:tomas.ulin@sun.com-20090217134419-5ha6xg4dpedrbmau)
(version source revid:tomas.ulin@sun.com-20090203133556-9rclp06ol19bmzs4) (merge vers:
5.1.32-ndb-6.3.22) (pib:6)

Pushed into 5.1.32-ndb-6.2.17 (revid:tomas.ulin@sun.com-20090217134216-5699eq74ws4oxa0j)
(version source revid:tomas.ulin@sun.com-20090201210519-vehobc4sy3g9s38e) (merge vers:
5.1.32-ndb-6.2.17) (pib:6)