MySQL Bugs: #39903: Tests that restart server fail sporadically on pushbuild

Bug #39903	Tests that restart server fail sporadically on pushbuild
Submitted:	7 Oct 2008 9:18	Modified:	30 Jan 2009 17:39
Reporter:	Sven Sandberg	Email Updates:
Status:	Closed	Impact on me:	None
Category:	MySQL Server: Tests	Severity:	S7 (Test Cases)
Version:	6.0-rpl	OS:	Any
Assigned to:	Serge Kozlov	CPU Architecture:	Any
Tags:	6.0-rpl-green, maria-no-logging, pushbuild, sporadic, test failure

Description:
maria-no-logging fails sporadically on pushbuild in 6.0-rpl. There are at least two common symptoms:

-------- FAILURE 1 --------
main.maria-no-logging                    [ fail ]

CURRENT_TEST: main.maria-no-logging
mysqltest: In included file "./include/maria_empty_logs.inc": At line 79: query 'use $default_db' failed: 2002: Can't connect to local MySQL server through socket '/dev/shm/pbtmp-n_mix-120/1/mysqld.1.sock' (2)

The result from queries just before the failure was:
set global maria_log_file_size=4294967295;
drop database if exists mysqltest;
create database mysqltest;
use mysqltest;
set global maria_checkpoint_interval=0;
create table t2 (a varchar(100)) engine=myisam;
insert into t2 select repeat('z',100);
insert into t2 select * from t2;
insert into t2 select * from t2;
insert into t2 select * from t2;
insert into t2 select * from t2;
insert into t2 select * from t2;
insert into t2 select * from t2;
* shut down mysqld, removed logs, restarted it

 - saving '/dev/shm/var-n_mix-120/1/log/main.maria-no-logging/' to '/dev/shm/var-n_mix-120/log/main.maria-no-logging/'

Retrying test, attempt(2/3)...

-------- END FAILURE 1 --------

-------- FAILURE 2 --------
main.maria-no-logging                    [ fail ]

CURRENT_TEST: main.maria-no-logging
mysqltest: In included file "./include/wait_until_connected_again.inc": At line 23: Server failed to restart

The result from queries just before the failure was:
< snip >
insert into t2 select * from t2;
insert into t2 select * from t2;
insert into t2 select * from t2;
insert into t2 select * from t2;
* shut down mysqld, removed logs, restarted it
create table t1 (a varchar(100)) engine=maria transactional=1;
show create table t1;
Table	Create Table
t1	CREATE TABLE `t1` (
  `a` varchar(100) DEFAULT NULL
) ENGINE=MARIA DEFAULT CHARSET=latin1 PAGE_CHECKSUM=1 TRANSACTIONAL=1
show engine maria logs;
Type	Name	Status
MARIA	Size        16384 maria_log.00000001	unknown
insert into t1 values('a');
insert into t1 select * from t2;
show engine maria logs;
Type	Name	Status
MARIA	Size        24576 maria_log.00000001	unknown
* shut down mysqld, removed logs, restarted it

More results from queries before failure can be found in /tmp/var-ps_row-7/2/log/maria-no-logging.log

 - saving '/tmp/var-ps_row-7/2/log/main.maria-no-logging/' to '/tmp/var-ps_row-7/log/main.maria-no-logging/'

Retrying test, attempt(2/3)...

-------- END FAILURE 2 --------

How to repeat:
Failure 1: https://intranet.mysql.com/secure/pushbuild/showpush.pl?dir=bzr_mysql-6.0-rpl&order=80 pb-valgrind/n_mix
xref: http://tinyurl.com/3wtg72

Failure 2: https://intranet.mysql.com/secure/pushbuild/showpush.pl?dir=bzr_mysql-6.0-rpl&order=80 sol10-amd64-a/ps_row
xref: http://tinyurl.com/4qcxey

For the second failure -- see Bug#37810

This is more general than maria-no-logging. It seems to happen for tests that have a server restart. See xref: http://tinyurl.com/6qudse

So it's probably a bug in the test framework.

Clarification: my last comment refers to failure 2.

I think that BUG#37810 is a different issue (the failure is on a different line in wait_until_connected_again). Failure 2 happens only in 6.0-rpl, whereas BUG#37810 happens in other trees too.

Failure 1 actually happens in other tests too, according to xref: http://tinyurl.com/6pby2g

It happens in:
  main.maria-autozerofill
  main.maria-no-logging
  main.maria-recovery
  main.maria-recovery2
  main.maria-recovery-bitmap

I think it sounds like a good idea to check number of connections and
then wait for this number at restart.  For now perhaps use include
file and later when we have new test framework, this could be done as
a perl function.

A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/59733

2738 Serge Kozlov	2008-11-25
      Bug#39903
      1. include/wait_for_status_var.inc: new primitive for waiting value of 
      a variable from SHOW STATUS.
      2. include/maria_empty_log.inc: added waiting of restoring all connections 
      to restarted server.

A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/59768

2738 Serge Kozlov	2008-11-25
      Bug#39903: Sporadically some maria tests cannot use default connection 
      after server restart. It happens because the connections to server
      did not restored properly and a test should check them before using. 
      Suggested solution added checking number of connected threads on server 
      before restart and then waiting this number on restarted server.
      1. include/wait_for_status_var.inc: new primitive for waiting value of
      a variable from SHOW STATUS.
      2. include/maria_empty_log.inc: added waiting of restoring all connections
      to restarted server.

A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/59807

2738 Serge Kozlov	2008-11-25
      Bug#39903: Sporadically some maria tests cannot use default connection
      after server restart. It happens because the connections to server
      did not restored properly and a test should check them before using.
      Suggested solution added checking number of connected threads on server
      before restart and then waiting this number on restarted server.
      1. include/wait_for_status_var.inc: new primitive for waiting value of
      a variable from SHOW STATUS.
      2. include/maria_empty_log.inc: added waiting of restoring all connections
      to restarted server.

A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/60024

2739 Serge Kozlov	2008-11-27
      Bug#39903: Sporadically some maria tests cannot use default connection 
      after server restart. It happens because the connections to server
      did not restored properly and a test should check them before using. 
      Suggested solution added checking number of connected threads on server 
      before restart and then waiting this number on restarted server.
      1. include/wait_for_status_var.inc: new primitive for waiting value of
      a variable from SHOW STATUS.
      2. include/maria_empty_log.inc: added waiting of restoring all connections
      to restarted server.

Pushed into 6.0.10-alpha (revid:luis.soares@sun.com-20090129165607-wiskabxm948yx463) (version source revid:luis.soares@sun.com-20090129163120-e2ntks4wgpqde6zt) (merge vers: 6.0.10-alpha) (pib:6)

Test case changes. No changelog entry needed.