MySQL Bugs: #38122: main.mysqlslap fails sporadically

Bug #38122	main.mysqlslap fails sporadically
Submitted:	15 Jul 2008 7:41	Modified:	29 Oct 2009 14:01
Reporter:	Alexander Nozdrin	Email Updates:
Status:	Closed	Impact on me:	None
Category:	Tools: Pushbuild	Severity:	S3 (Non-critical)
Version:	1	OS:	Any
Assigned to:	Daniel Fischer	CPU Architecture:	Any
Tags:	hw, pb1, pushbuild, sporadic, test failure

Description:
main.mysqlslap                 [ fail ]

/data0/pushbuild/pb/bzr_mysql-6.0/26/mysql-6.0.6-alpha-pb26/client/mysqlslap: Cannot run query INSERT INTO t1 VALUES (NULL,2126558185,773273718,'Y88N07to033m2XEIzHzMJsHuXTL2GeCheC4WJqIClDqol4ycMpqZh7K4cZytBBQhNMBb4LnqadWBZSfvITLzdZw56uqH5GqTftwj6jZYC3klONZalmPEJLJPGAaLIO','ip9eZFDynPuN3AnSkOE9ePXxinzG0IYjzhLYobNbs9znJeFv2L4YmRTS6Lc7X0qWZ3vLWApFJo2WIAPKXLsIhDenQ7ux7cKhYFu2Pj9cNCyhMFtlSd39raxHhj7gKR','Dsa0mrjwR70PgEYaz0BvAuLz4uiBa2Wgm4gPvFDEMDXvR3TI3gD5RQ4XKnivhCRK87l3My9d3gYMbtgePmcsCgNZRxK11nL9m640wDTzltLKm1r4fRws7CHR9TSBGp') ERROR : Disk is full writing './mysqlslap/t1.MYD' (Errcode: 28). Waiting for someone to free space... Retry in 60 secs
mysqltest: At line 45: command "$MYSQL_SLAP --silent --concurrency=5 --iterations=1 --number-int-cols=2 --number-char-cols=3 --auto-generate-sql --auto-generate-sql-add-autoincrement --auto-generate-sql-load-type=write --timer-length=10" failed

The result from queries just before the failure was:
< snip >
insert into t2 values ('test', 'test2');
SET AUTOCOMMIT=0;
SHOW TABLES;
SET AUTOCOMMIT=0;
select * from t1;
COMMIT;
select * from t2;
COMMIT;
select * from t1;
COMMIT;
select * from t2;
COMMIT;
select * from t1;
COMMIT;
select * from t2;
COMMIT;
COMMIT;
SHOW TABLES;
DROP SCHEMA IF EXISTS `mysqlslap`;
exec of '/data0/pushbuild/pb/bzr_mysql-6.0/26/mysql-6.0.6-alpha-pb26/client/mysqlslap -uroot --port=11320 --socket=/dev/shm/pbtmp-ps_stm-132/master.sock --password=  --silent --concurrency=5 --iterations=1 --number-int-cols=2 --number-char-cols=3 --auto-generate-sql --auto-generate-sql-add-autoincrement --auto-generate-sql-load-type=write --timer-length=10' failed, error: 256, status: 1, errno: 0

How to repeat:
https://intranet.mysql.com/secure/pushbuild/xref.pl?testname=main.mysqlslap

Sorry if I'm missing something, however the error seems to imply that you are out of disk space:

Error: disk is full

BJ Dierkes is right.
mysqlslap fails because the disk is full.

Some statistics based on a query with xref
     Test name: main.mysqlslap
     Test output match (LIKE): Disk is full
on our internal Pushbuild database gives the
following distribution of matches 
- 26 at all
- 23 MySQL 6.0, 3 MySQL 5.1
- 25 Linux 64 Bit on sapsrv2 (use of tmpfs for 
     filesystem containing VARDIR/TMPDIR)
  1 Solaris 10 box (no info about FS used)
- all types of replication and ps/no ps protocol
  appear
- failing test runs between 2007-12 and 2008-07
- At least sometimes not only mysqlslap fails
  with a message about "disk is full".

Some statistics based on a query with xref
     Test output match (LIKE): Disk is full
on our internal Pushbuild database gives the
following distribution of matches
- 722 at all
- 110 sapsrv2, 26 debx86-b, 206 vm-win2003-32-a,
  93 sapsrv1
- boxes which appear most often within the last
  2 months: sapsrv2, vm-win2003-32-a, sapsrv1

Some experiments on my Linux box (MySQL 5.1):
- The maximum disk space consumption of the main test
  stuff (VARDIR,TMPDIR) was ~ 12.5 MB which is around
  the same amount for the "simple" test alias.
- The maximum virtual memory size
  (ps -e -o vsize,command | egrep -e "VSZ|lt-mysqlslap")
  of the mysqlslap process was ~ 128 MB.
- It does not look like the mysqlslap memory consumption
  increases with the lifetime of the process. 

Conclusions:
- There seems to be no correlation to a certain type
  of test (ps, replication method, ...) or OS
  (We have to take the huge amount of successful runs
  on Linux 64 Bit in general into account).
- The observed maximum space consumption of ~ 12.5 MB
  for test related data (files for protocols/tables,..)
  do not indicate that the test is too "greedy".
- IMHO the maximum virtual memory size of the mysqlslap 
  process of ~ 128 MB is the most critical part of
  this test. But it could not be said that this process
  is not acceptable "greedy". There are other tests which
  also fail sporadically with Disk is full.
- It looks like the bug reported is a temporary bad
  state on the box used for testing.
  The testing box sapsrv2 appears quite often.
  I assume this box suffers sometimes from a combination
  of heavy load and too small swap space.

I propose one or a combination of the following actions
for all testing boxes showing "disk is full" problems
1. Increase the swap space and maybe add RAM
   to sapsrv2 and sapsrv1
2. Reduce the use of tmpfs (some other parallel tests
   might store a lot of data in VARDIR/TMPDIR located
   in tmpfs)
   on sapsrv2 and sapsrv1
3. Decrease the load (parallel processes consuming
   significant amount of virtual memory)
   on sapsrv2 and sapsrv1
It could be that the problem with sapsrv1 is already
fixed because the last problem was 2008-06-11.
4. I do not know what to do in case of vm-win2003-32-a.

I will set
- Category to "Pushbuild"
  We have unfortunately no category
  "Pushbuild - testing box configuration mismatch"
- Lead to "Daniel Fisher"
- Remove the assignment to me

Reported for server versions 5.1,6.0.

Closing because we don't use this host anymore.