MySQL Bugs: #42003: tests missing the disconnect of connections <> default

Bug #42003	tests missing the disconnect of connections <> default
Submitted:	9 Jan 2009 18:37	Modified:	18 Mar 2009 14:40
Reporter:	Matthias Leich	Email Updates:
Status:	Closed	Impact on me:	None
Category:	MySQL Server: Tests	Severity:	S3 (Non-critical)
Version:	5.0,5.1,6.0	OS:	Any
Assigned to:	Matthias Leich	CPU Architecture:	Any
Tags:	pushbuild, sporadic, test failure

Description:
We have many tests which do not close at end all 
connections created during test execution.

There is some probability that "delayed" actions
of these connections have an impact on the outcome
of the succeeding test like
- too late entries in logs
  Example from bug http://bugs.mysql.com/bug.php?id=40377
  CURRENT_TEST: main.log_state
  --- bzr_mysql-6.0-rpl/.../r/log_state.result 
  +++ bzr_mysql-6.0-rpl/.../r/log_state.reject
  @@ -25,6 +25,7 @@
  event_time user_host ... command_type	argument
  TIMESTAMP  USER_HOST ... Query	create table t1(f1 int)
  TIMESTAMP  USER_HOST ... Query	select * from mysql.general_log
 +TIMESTAMP  USER_HOST ... Quit	
  TIMESTAMP  USER_HOST ... Query	set global general_log= OFF
  set global general_log= ON;
  flush logs;

  BTW: The end of t/log_state.test is
       #
       # Cleanup (must be done last to avoid delayed 'Quit' message 
         in general log)
       disconnect con1;
- unexpected entries within the processlist or less
  less free connections available than expected
  This can happen in case of active events where this
  has to be avoided by waiting till the event executor
  disappears.
  But I assume this can also happen in case of
  a) "DELAYED" SQL command          or
  b) "send" SQL command without following "reap" (*)
  + end of the script without disconnect.
  (*) I reached 1040: Too many connections
      in experiments.

The attached file "check.lst" contains a list of tests
where it might be that some disconnect at test end is
missing.
- source was mysql-5.1-bugtem
- "egrep" "connect|disconnect" t/* | <other filters>
- some editing
It is to be expected that this list
- is incomplete
- contains false positives
because connect/disconnects done within sourced files
(example: include/*.inc) are ignored. But it is a good
starting point.

How to repeat:
Just look into the tests mentioned within the list.

The suspicious tests

Attachment: check.lst (application/octet-stream, text), 10.34 KiB.

A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/64060

2735 Matthias Leich	2009-01-26
      1. Slice of fix for Bug#42003 tests missing the disconnect of connections <> default
            - If missing: add "disconnect <session>"
            - If physical disconnect of non "default" sessions is not finished
              at test end: add routine which waits till this happened
      + additional improvements
            - remove superfluous files created by the test
            - replace error numbers by error names
            - remove trailing spaces, replace tabs by spaces
            - unify writing of bugs within comments
            - correct comments
            - minor changes of formatting
            ...

Hi Patrick,

thank you for the review.

- Special thanks for the hint about open_ssl1.
- Bug#2005 with wrong word "comparion"
  mysqldump.test contains the exact bug title.
  I will correct the title within bugsdb
  and the comment within the test.
- Bug#12597 with wrong word "wronlgy"
  fix like above
- grant2: subtest around line 500 modified

Regards

Matthias

A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/64526

2735 Matthias Leich	2009-01-29
      1. Slice of fix for Bug#42003 tests missing the disconnect of connections <> default
         - If missing: add "disconnect <session>"
         - If physical disconnect of non "default" sessions is not finished
           at test end: add routine which waits till this happened
      + additional improvements
        - remove superfluous files created by the test
        - replace error numbers by error names
        - remove trailing spaces, replace tabs by spaces
        - unify writing of bugs within comments
        - correct comments
        - minor changes of formatting
      Modifications according to the code review are included.
      Fixed tests:
      grant2
      grant3
      lock_tables_lost_commit
      mysqldump
      openssl_1
      outfile

The 1. slice of fixes is approved.
I reset the reviewer check buttons for
the review of the second slice of fixwes.

A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/64618

2736 Matthias Leich	2009-01-30
      2. Slice of fix for Bug#42003 tests missing the disconnect of connections <> default
         - If missing: add "disconnect <session>"
         - If physical disconnect of non "default" sessions is not finished
           at test end: add routine which waits till this happened
      + additional improvements
        - remove superfluous files created by the test
        - replace error numbers by error names
        - remove trailing spaces, replace tabs by spaces
        - unify writing of bugs within comments
        - correct comments
        - minor changes of formatting
      Fixed tests:
      backup
      check
      compress
      grant
      information_schema
      multi_update
      overflow
      packet
      query_cache_not_embedded
      sp-threads
      subselect
      synchronization
      timezone_grant

A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/64938

2735 Matthias Leich	2009-02-02
      1. Slice of fix for Bug#42003 tests missing the disconnect of connections <> default
         - If missing: add "disconnect <session>"
         - If physical disconnect of non "default" sessions is not finished
           at test end: add routine which waits till this happened
      + additional improvements like
        - remove superfluous files created by the test
        - replace error numbers by error names
        - remove trailing spaces, replace tabs by spaces
        - unify writing of bugs within comments
        - correct comments
        - minor changes of formatting
      Modifications according to the code review are included.
      Fixed tests:
      grant2
      grant3
      lock_tables_lost_commit
      mysqldump
      openssl_1
      outfile

Slice 1 which contains fixes for
      grant2
      grant3
      lock_tables_lost_commit
      mysqldump
      openssl_1
      outfile
was pushed to
mysql-6.0-bugteam
mysql-5.1-bugteam
mysql-5.0-bugteam

A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/65405

2737 Matthias Leich	2009-02-05
      2. Slice of fix for Bug#42003 tests missing the disconnect of connections <> default
         - If missing: add "disconnect <session>"
         - If physical disconnect of non "default" sessions is not finished
           at test end: add routine which waits till this happened
      + additional improvements
        - remove superfluous files created by the test
        - replace error numbers by error names
        - remove trailing spaces, replace tabs by spaces
        - unify writing of bugs within comments
        - correct comments
        - minor changes of formatting
      Fixed tests:
        backup
        check
        compress
        grant
        information_schema
        multi_update
        overflow
        packet
        query_cache_not_embedded
        sp-threads
        subselect
        synchronization
        timezone_grant

Pushed into 5.0.78 (revid:joro@sun.com-20090206133929-ybpbkkox9wkptopb) (version source revid:matthias.leich@sun.com-20090202212025-c54gfn1ob3o6zrw7) (merge vers: 5.0.78) (pib:6)

A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/65658

2740 Matthias Leich	2009-02-09
      This belongs to the fix for
      Bug#42003 tests missing the disconnect of connections <> default
      second slice
      Content:
      1. wait_until_count_sessions.inc
         - One PB run of a test using this routine failed because
           5 seconds timeout were exceeded. Although I have some doubts
           if the assigned timeout was really too small, I increase the
           value to 10. We waste the additional 5 seconds only if the
           tests fails anyway.
         - Print the content of the PROCESSLIST if the poll routine fails
      2. minor improvements of formatting
      3. query_cache_notembedded:
         Activate the wait_until_count_sessions.inc routine which was
         unfortunately forgotten in the changeset before.

Pushed into 5.1.32 (revid:davi.arnaut@sun.com-20090209214102-gj3sb3ujpnvpiy4c) (version source revid:davi.arnaut@sun.com-20090209214102-gj3sb3ujpnvpiy4c) (merge vers: 5.1.32) (pib:6)

Hi Horst,

thank you for your review.

1. The sleeps in multi_update:
   I do not want to replace them in 5.0.
   The situation in 5.0 is unfortunate. We cannot
   pull the information needed via a SELECT from
   information_schema.processlist because this table
   does not exist.
   Of course a very sophisticated routine which
   processes the SHOW PROCESSLIST output with
   query_get_value could be developed but
   - this is very expensive compared to the advantages we get
   - there are a lot more tests which are faced by
     similar prtoblems and just use a "sleep"
   The sleeps within multi_update will replaced by
   SELECTs on information_schema.processlist within
   wait routines when merging 5.0 to 5.1.

2. "information_schema", "multi_update" and "subselect"
   without the routines which ensure that all additional
   sessions have been really removed:
   A have my own very aggressive and to some unknown
   extend over-sensitive check which simulates on
   overloaded testing box.
   "information_schema", "multi_update" and "subselect"
   never failed within this test. IMHO the reason for
   their "success" is that the disconnects of parallel
   sessions are far away of the test end. This means
   it is extreme improbable that the server does not
   finish the removal of these sessions during the period
   where the remaining statements of the test are executed.

   All other tests which failed when my aggressive check
   was applied were treated 
   
   Do you insist that I add the routines to make the test
   total "waterproof" ? 

Regards

Matthias

Slice 2 which contains fixes for
        backup
        check
        compress
        grant
        information_schema
        multi_update
        overflow
        packet
        query_cache_not_embedded
        sp-threads
        subselect
        synchronization
        timezone_grant
was pushed to
mysql-6.0-bugteam
mysql-5.1-bugteam
mysql-5.0-bugteam

I reset state to "in progress" and the check
buttons of the reviewers because the last slice
of the fixes has to be implemented.

Pushed into 6.0.10-alpha (revid:matthias.leich@sun.com-20090212211028-y72faag15q3z3szy) (version source revid:matthias.leich@sun.com-20090211164850-o3cawmmsrc789ajs) (merge vers: 6.0.10-alpha) (pib:6)

Pushed into 5.1.32-ndb-6.3.23 (revid:tomas.ulin@sun.com-20090217131017-6u8qz1edkjfiobef) (version source revid:tomas.ulin@sun.com-20090216083408-rmvyaxjt6mk8sg1y) (merge vers: 5.1.32-ndb-6.3.23) (pib:6)

Pushed into 5.1.32-ndb-6.4.3 (revid:tomas.ulin@sun.com-20090217134419-5ha6xg4dpedrbmau) (version source revid:tomas.ulin@sun.com-20090216083646-m8st11oj1hhfuuh5) (merge vers: 5.1.32-ndb-6.4.3) (pib:6)

Pushed into 5.1.32-ndb-6.2.17 (revid:tomas.ulin@sun.com-20090217134216-5699eq74ws4oxa0j) (version source revid:tomas.ulin@sun.com-20090211111208-wf0acl7c1vl5653e) (merge vers: 5.1.32-ndb-6.2.17) (pib:6)

Last slice of fixes

Attachment: 42003.patch (text/x-patch), 162.54 KiB.

The attached file 42003.patch
contains the last slice of fixes.

A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/68144

A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/68384

A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/68509

A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/68557

Pushed into 5.0.79 (revid:joro@sun.com-20090309135922-a0di9ebkxoj4d4wv) (version source revid:matthias.leich@sun.com-20090209205240-wcp5ujjcmlfpztli) (merge vers: 5.0.79) (pib:6)

The last slice contains fixes for the following tests
  alter_table-big
  connect
  consistent_snapshot
  dirty_close
  flush_block_commit_notembedded
  flush_block_commit
  flush_read_lock_kill
  init_connect
  lock_multi
  mysqlbinlog
  mysqltest
  read_only
  show_check
  skip_name_resolve
  sp_notembedded
  sp-security
  ssl-big
  ssl_compress
  ssl
  status
  user_limits
  view_grant
  view
  wait_timeout
  xa
and a fix for
  Bug#43114 wait_until_count_sessions too restrictive,
            random PB failures

Pushed to
mysql-6.0-bugteam
mysql-5.1-bugteam
mysql-5.0-bugteam

Pushed into 5.1.33 (revid:joro@sun.com-20090313111355-7bsi1hgkvrg8pdds) (version source revid:zhou.li@sun.com-20090311061050-ihp0g77znonq1tuq) (merge vers: 5.1.33) (pib:6)

Pushed into 5.0.80 (revid:chad@mysql.com-20090312173213-viivy35mb79n1zlg) (version source revid:chad@mysql.com-20090312173213-viivy35mb79n1zlg) (merge vers: 5.0.80) (pib:6)

Test case changes. No changelog entry needed.

Setting report to NDI pending push into 6.0.x.

Pushed into 6.0.11-alpha (revid:joro@sun.com-20090318122208-1b5kvg6zeb4hxwp9) (version source revid:matthias.leich@sun.com-20090306203352-rf8dh80l21cls5t2) (merge vers: 6.0.10-alpha) (pib:6)

Test case changes. No changelog entry needed.

Pushed into 5.1.34-ndb-6.2.18 (revid:jonas@mysql.com-20090508185236-p9b3as7qyauybefl) (version source revid:jonas@mysql.com-20090508100057-30ote4xggi4nq14v) (merge vers: 5.1.33-ndb-6.2.18) (pib:6)

Pushed into 5.1.34-ndb-6.3.25 (revid:jonas@mysql.com-20090509063138-1u3q3v09wnn2txyt) (version source revid:jonas@mysql.com-20090508175813-s6yele2z3oh6o99z) (merge vers: 5.1.33-ndb-6.3.25) (pib:6)

Pushed into 5.1.34-ndb-7.0.6 (revid:jonas@mysql.com-20090509154927-im9a7g846c6u1hzc) (version source revid:jonas@mysql.com-20090509073226-09bljakh9eppogec) (merge vers: 5.1.33-ndb-7.0.6) (pib:6)