MySQL Bugs: #22379: im_daemon_life_cycle.test fails on merge of 5.1 -> 5.1-engines

Bug #22379	im_daemon_life_cycle.test fails on merge of 5.1 -> 5.1-engines
Submitted:	15 Sep 2006 7:44	Modified:	4 Oct 2006 19:22
Reporter:	Ingo Strüwing	Email Updates:
Status:	Closed	Impact on me:	None
Category:	Instance Manager	Severity:	S1 (Critical)
Version:	5.0 + 5.1	OS:	Linux (Linux)
Assigned to:	Magnus Blåudd	CPU Architecture:	Any

Description:
im_cmd_line                    [ pass ]           2112
mysql-test-run: WARNING: check_expected_crash_and_restart couldn't find an entry for pid: 21818
im_daemon_life_cycle           [ pass ]           3038
Warning: [Thu Sep 14 23:12:57 2006] [im_daemon_life_cycle] Instance Manager did not shutdown gracefully.
Warning: [Thu Sep 14 23:12:57 2006] Mysqld is accepting connections on port 13048, but there is no process information.
mysql-test-run: *** ERROR: Can not kill Instance Manager or its children.
make: *** [test-ns] Error 1

Checked it twice to exclude a temporary flaw.

It might be a test script problem instead of an instance manager problem. I did not find a running instance manager process after the test. So the check by the script might be incorrect. If so, please change category.

How to repeat:
Log in on 'production'.
Stay at your home directory to use the local disk.
Do not work on a nfs disk.
bk clone bk-internal:/home/bk/mysql-5.1-engines mysql-5.1-engines
cd mysql-5.1-engines
Pull bk-internal:/home/bk/mysql-5.1
BUILD/compile-pentium-debug-max
make test

Can be seen in most clones.
The significant error message is "mysql-test-run: *** ERROR: Can not kill Instance Manager or its children."

This is cause by the mysqld started by im is still alive but there is no pid file for it.

A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/12354

ChangeSet@1.2348, 2006-09-21 17:56:36+02:00, msvensson@shellback.(none) +3 -0
  Bug #22379 im_daemon_life_cycle.test fails on merge of 5.1 -> 5.1-engines
   - Remove race situations that occur when removing pidfiles. 
     Primarily each process should remove it's own pidfile, secondly
     it should be removed by the process that created it and _only_ if it's
     certain the process is dead. Third, mysql-test-run.pl will remove
     the pidfile when process has been killed.

A check that fscanf returns 1 is also needed to avoid race, see below patch:

[msvensson@shellback instance-manager]$ bk diffs -u instance_options.cc
===== instance_options.cc 1.36 vs edited =====
--- 1.36/server-tools/instance-manager/instance_options.cc      2006-09-21 19:19:52 +02:00
+++ edited/instance_options.cc  2006-09-21 18:53:28 +02:00
@@ -406,7 +406,8 @@
   {
     pid_t pid;

-    fscanf(pid_file_stream, "%i", &pid);
+    if (fscanf(pid_file_stream, "%i", &pid) != 1)
+      pid= 0;
     my_fclose(pid_file_stream, MYF(0));
     return pid;
   }

A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/12360

ChangeSet@1.2281, 2006-09-21 20:23:12+02:00, msvensson@shellback.(none) +4 -0
  Bug #22379 im_daemon_life_cycle.test fails on merge of 5.1 -> 5.1-engines
  - Remove race situations that occur when removing pidfiles. 
    Primarily each process should remove its own pidfile, secondly
    it should be removed by the process that created it and _only_ if it's
    certain the process is dead. Third, mysql-test-run.pl will remove
    the pidfile when process has been killed.

A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/12405

ChangeSet@1.2355, 2006-09-22 15:54:09+02:00, msvensson@neptunus.(none) +1 -0
  Bug#22379 im_daemon_life_cycle.test fails on merge of 5.1 -> 5.1-engines

A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/12407

ChangeSet@1.2286, 2006-09-22 16:14:23+02:00, msvensson@neptunus.(none) +1 -0
  Bug#22379  im_daemon_life_cycle.test fails on merge of 5.1 -> 5.1-engines
   - Set state to STARTING before calling instance->start()
   - Check that pidfile has been created before changing STARTING => STARTED

A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/12457

ChangeSet@1.2245, 2006-09-25 09:28:52+02:00, msvensson@neptunus.(none) +3 -0
  Bug #22379 im_daemon_life_cycle.test fails on merge of 5.1 -> 5.1-engines
  Remove race situations that occur when removing pidfiles. Primarily each process should remove its own
  pidfile, secondly it should be removed by the process that created it and _only_ if it's
  certain the process is dead. Third, mysql-test-run.pl will remove the pidfile when process has been killed.
  - Set state of an instance to STARTING _before_ calling instance->start()
  - Check that pidfile of instance has been created before changing STARTING => STARTED
  - Only remove the pidfile if IM kills an instance with SIGKILL, otherwise the instance will remove it itself

pushed to 5.1.12

Pushed to 5.0.27

Noted in 5.0.27, 5.1.12 changelogs.

Instance Manager had a race condition involving mysqld PID file
removal.

A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/14267

ChangeSet@1.2325, 2006-10-24 18:25:24+04:00, anozdrin@alik. +1 -0
  Enable im_daemon_life_cycle.imtest, since BUG#22379 is closed.

The 5.0.x fix is in 5.0.30.