Bug #37003 Tests sporadically crashes with embedded server
Submitted: 27 May 2008 12:38 Modified: 4 Aug 2008 16:26
Reporter: Alexander Nozdrin Email Updates:
Status: Closed Impact on me:
None 
Category:MySQL Server: Embedded Library ( libmysqld ) Severity:S1 (Critical)
Version:5.1 OS:Any
Assigned to: Davi Arnaut CPU Architecture:Any
Tags: pushbuild, sporadic, test failure

[27 May 2008 12:38] Alexander Nozdrin
Description:
Tests running on the embedded server crash randomly.

Usual error message is as follows:
---------------------------------------------
<test name> [ fail ]

ERROR: mysqltest returned unexpected code 139, it has probably crashed

Stopping All Servers
Restoring snapshot of databases
Resuming Tests
---------------------------------------------

In 6.0 tests are crashing more heavily.

Probably related: Bug#31850 (Test crashes in "embedded" server).

How to repeat:
https://intranet.mysql.com/secure/pushbuild/showpush.pl?dir=mysql-5.1&order=1629
https://intranet.mysql.com/secure/pushbuild/xref.pl?testname=main.sql_low_priority_updates...
[27 May 2008 17:45] Bugs System
A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/47102
[28 May 2008 22:38] Davi Arnaut
This bug is hard to investigate because of a shortcoming in our testing of embedded builds. Currently we only have a message like "it has probably crashed" when something fails (crash/assert/abort) the core file is our only hope to find out where it crashed, but core files are disabled on some hosts. We probably need to move the signal handling and stacktrace facilities of mysqld to mysys so we can use it in mysqltest (and others) to generate traces.

Another problem is a crash (embedded server) related to sending commands to closed connections (using the connection handle after is has been closed by the application). Arguably, this could be a mysqltest bug but a few tests expect this behavior and this works for non-embedded builds:

connect (con1,localhost,root,,);
select 1;
disconnect con1;
select 1;
[5 Jun 2008 18:31] Bugs System
A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/47494

2660 Davi Arnaut	2008-06-05
      Bug#37003 Tests sporadically crashes with embedded server
      
      The problem was that when a embedded linked version of mysqltest
      crashed there was no way to obtain a stack trace if no core file
      is available. Another problem is that the embedded version of
      libmysql was not behaving (crash) the same as the non-embedded with
      respect to sending commands to a explicitly closed connection.
      
      The solution is to generate a mysqltest's stack trace on crash
      and to enable "reconnect" if the connection handle was explicitly
      closed so the behavior matches the non-embedded one.
[18 Jun 2008 16:11] Bugs System
A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/48107

2667 Davi Arnaut	2008-06-18
      Bug#37003 Tests sporadically crashes with embedded server
      
      The problem was that when a embedded linked version of mysqltest
      crashed there was no way to obtain a stack trace if no core file
      is available. Another problem is that the embedded version of
      libmysql was not behaving (crash) the same as the non-embedded with
      respect to sending commands to a explicitly closed connection.
      
      The solution is to generate a mysqltest's stack trace on crash
      and to enable "reconnect" if the connection handle was explicitly
      closed so the behavior matches the non-embedded one.
[18 Jun 2008 16:22] Davi Arnaut
Queued to 5.1-bugteam
[19 Jun 2008 14:05] Bugs System
A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/48175

2669 Davi Arnaut	2008-06-19
      Bug#37003 Tests sporadically crashes with embedded server
      
      Another problem is that the backtrace facility wasn't being
      enabled for non-Linux targets even if the target OS has the
      backtrace functions. Also, the stacktrace functions inside
      mysqltest were being used without proper checks for their
      presence in the build.
[19 Jun 2008 14:12] Bugs System
A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/48177

2669 Davi Arnaut	2008-06-19
      Bug#37003 Tests sporadically crashes with embedded server
      
      Another problem is that the backtrace facility wasn't being
      enabled for non-Linux targets even if the target OS has the
      backtrace functions. Also, the stacktrace functions inside
      mysqltest were being used without proper checks for their
      presence in the build.
[20 Jun 2008 11:36] Bugs System
A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/48230
[20 Jun 2008 12:23] Bugs System
A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/48237
[20 Jun 2008 14:24] Bugs System
A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/48244
[4 Jul 2008 8:58] Alexander Nozdrin
The patch was pushed to 6.0.

There are however new failures:
https://intranet.mysql.com/secure/pushbuild/showpush.pl?dir=bzr_mysql-6.0&order=24

Additional investigation/fix needed. Moving to Verified.
[28 Jul 2008 16:50] Bugs System
Pushed into 5.1.27  (revid:joerg@mysql.com-20080626161101-h5hsaluwtf05xdr2) (version source revid:joerg@mysql.com-20080626161101-h5hsaluwtf05xdr2) (pib:3)
[4 Aug 2008 16:26] Paul DuBois
Test case changes. No changelog entry needed.
[4 Aug 2008 18:04] Paul DuBois
Noted in 5.1.27, 6.0.7 changelogs.

mysqltest now installs signal handlers and generates a stack trace if
it crashes.
[6 Aug 2008 14:18] Bugs System
A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/51014

2683 Kristofer Pettersson	2008-08-06 [merge]
      Manual merge 5.0->5.1
[7 Aug 2008 18:39] Bugs System
A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/51138

2744 Marc Alff	2008-08-07 [merge]
      Merge mysql-6.0-bugteam -> local bugfix branch
[12 Aug 2008 14:44] Bugs System
Pushed into 6.0.7-alpha  (revid:davi.arnaut@sun.com-20080801151006-kjthw3lgkkb0u2a8) (version source revid:davi.arnaut@sun.com-20080812141852-8e6knbqclpfd8irn) (pib:3)
[12 Aug 2008 15:13] Bugs System
Pushed into 5.1.28  (revid:davi.arnaut@sun.com-20080801151006-kjthw3lgkkb0u2a8) (version source revid:davi.arnaut@sun.com-20080812142843-he05ncsggstbn57z) (pib:3)
[28 Aug 2008 20:16] Bugs System
Pushed into 6.0.7-alpha  (revid:cbell@mysql.com-20080822132131-uveo6wiuecy6m2b8) (version source revid:cbell@mysql.com-20080822132131-uveo6wiuecy6m2b8) (pib:3)
[14 Sep 2008 1:25] Bugs System
Pushed into 6.0.7-alpha  (revid:davi.arnaut@sun.com-20080801151006-kjthw3lgkkb0u2a8) (version source revid:vvaintroub@mysql.com-20080804094710-jb2qpqxpf2ir2gf3) (pib:3)