Bug #37003 Tests sporadically crashes with embedded server
Submitted: 27 May 2008 14:38 Modified: 4 Aug 2008 18:26
Reporter: Alexander Nozdrin
Status: Closed
Category:libmysqld Severity:S1 (Critical)
Version:5.1 OS:Any
Assigned to: Davi Arnaut Target Version:5.1+
Tags: sporadic, test failure, pushbuild
Triage: D1 (Critical)

[27 May 2008 14:38] Alexander Nozdrin
Description:
Tests running on the embedded server crash randomly.

Usual error message is as follows:
---------------------------------------------
<test name> [ fail ]

ERROR: mysqltest returned unexpected code 139, it has probably crashed

Stopping All Servers
Restoring snapshot of databases
Resuming Tests
---------------------------------------------

In 6.0 tests are crashing more heavily.

Probably related: Bug#31850 (Test crashes in "embedded" server).

How to repeat:
https://intranet.mysql.com/secure/pushbuild/showpush.pl?dir=mysql-5.1&order=1629
https://intranet.mysql.com/secure/pushbuild/xref.pl?testname=main.sql_low_priority_updates...
[27 May 2008 19:45] Bugs System
A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/47102
[29 May 2008 0:38] Davi Arnaut
This bug is hard to investigate because of a shortcoming in our testing of embedded
builds. Currently we only have a message like "it has probably crashed" when something
fails (crash/assert/abort) the core file is our only hope to find out where it crashed,
but core files are disabled on some hosts. We probably need to move the signal handling
and stacktrace facilities of mysqld to mysys so we can use it in mysqltest (and others)
to generate traces.

Another problem is a crash (embedded server) related to sending commands to closed
connections (using the connection handle after is has been closed by the application).
Arguably, this could be a mysqltest bug but a few tests expect this behavior and this
works for non-embedded builds:

connect (con1,localhost,root,,);
select 1;
disconnect con1;
select 1;
[5 Jun 2008 20:31] Bugs System
A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/47494

2660 Davi Arnaut	2008-06-05
      Bug#37003 Tests sporadically crashes with embedded server
      
      The problem was that when a embedded linked version of mysqltest
      crashed there was no way to obtain a stack trace if no core file
      is available. Another problem is that the embedded version of
      libmysql was not behaving (crash) the same as the non-embedded with
      respect to sending commands to a explicitly closed connection.
      
      The solution is to generate a mysqltest's stack trace on crash
      and to enable "reconnect" if the connection handle was explicitly
      closed so the behavior matches the non-embedded one.
[18 Jun 2008 18:11] Bugs System
A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/48107

2667 Davi Arnaut	2008-06-18
      Bug#37003 Tests sporadically crashes with embedded server
      
      The problem was that when a embedded linked version of mysqltest
      crashed there was no way to obtain a stack trace if no core file
      is available. Another problem is that the embedded version of
      libmysql was not behaving (crash) the same as the non-embedded with
      respect to sending commands to a explicitly closed connection.
      
      The solution is to generate a mysqltest's stack trace on crash
      and to enable "reconnect" if the connection handle was explicitly
      closed so the behavior matches the non-embedded one.
[18 Jun 2008 18:22] Davi Arnaut
Queued to 5.1-bugteam
[19 Jun 2008 16:05] Bugs System
A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/48175

2669 Davi Arnaut	2008-06-19
      Bug#37003 Tests sporadically crashes with embedded server
      
      Another problem is that the backtrace facility wasn't being
      enabled for non-Linux targets even if the target OS has the
      backtrace functions. Also, the stacktrace functions inside
      mysqltest were being used without proper checks for their
      presence in the build.
[19 Jun 2008 16:12] Bugs System
A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/48177

2669 Davi Arnaut	2008-06-19
      Bug#37003 Tests sporadically crashes with embedded server
      
      Another problem is that the backtrace facility wasn't being
      enabled for non-Linux targets even if the target OS has the
      backtrace functions. Also, the stacktrace functions inside
      mysqltest were being used without proper checks for their
      presence in the build.
[20 Jun 2008 13:36] Bugs System
A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/48230
[20 Jun 2008 14:23] Bugs System
A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/48237
[20 Jun 2008 16:24] Bugs System
A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/48244
[4 Jul 2008 10:58] Alexander Nozdrin
The patch was pushed to 6.0.

There are however new failures:
https://intranet.mysql.com/secure/pushbuild/showpush.pl?dir=bzr_mysql-6.0&order=24

Additional investigation/fix needed. Moving to Verified.
[28 Jul 2008 18:50] Bugs System
Pushed into 5.1.27  (revid:joerg@mysql.com-20080626161101-h5hsaluwtf05xdr2) (version
source revid:joerg@mysql.com-20080626161101-h5hsaluwtf05xdr2) (pib:3)
[4 Aug 2008 18:26] Paul DuBois
Test case changes. No changelog entry needed.
[4 Aug 2008 20:04] Paul DuBois
Noted in 5.1.27, 6.0.7 changelogs.

mysqltest now installs signal handlers and generates a stack trace if
it crashes.
[6 Aug 2008 16:18] Bugs System
A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/51014

2683 Kristofer Pettersson	2008-08-06 [merge]
      Manual merge 5.0->5.1
[7 Aug 2008 20:39] Bugs System
A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/51138

2744 Marc Alff	2008-08-07 [merge]
      Merge mysql-6.0-bugteam -> local bugfix branch
[12 Aug 2008 16:44] Bugs System
Pushed into 6.0.7-alpha  (revid:davi.arnaut@sun.com-20080801151006-kjthw3lgkkb0u2a8)
(version source revid:davi.arnaut@sun.com-20080812141852-8e6knbqclpfd8irn) (pib:3)
[12 Aug 2008 17:13] Bugs System
Pushed into 5.1.28  (revid:davi.arnaut@sun.com-20080801151006-kjthw3lgkkb0u2a8) (version
source revid:davi.arnaut@sun.com-20080812142843-he05ncsggstbn57z) (pib:3)
[28 Aug 2008 22:16] Bugs System
Pushed into 6.0.7-alpha  (revid:cbell@mysql.com-20080822132131-uveo6wiuecy6m2b8) (version
source revid:cbell@mysql.com-20080822132131-uveo6wiuecy6m2b8) (pib:3)
[14 Sep 2008 3:25] Bugs System
Pushed into 6.0.7-alpha  (revid:davi.arnaut@sun.com-20080801151006-kjthw3lgkkb0u2a8)
(version source revid:vvaintroub@mysql.com-20080804094710-jb2qpqxpf2ir2gf3) (pib:3)