MySQL Bugs: #40399: Please make mtr print stack trace after every failure

Bug #40399	Please make mtr print stack trace after every failure
Submitted:	29 Oct 2008 18:29	Modified:	30 Jan 18:46
Reporter:	Sven Sandberg
Status:	Closed
Category:	Tests	Severity:	S2 (Serious)
Version:	5.1	OS:	Any
Assigned to:	Bjorn Munch	Target Version:
Tags:	mtr
Triage:	Triaged: D5 (Feature request)

Description:
There are a number of common test failures that cannot be debugged. The error messages
can be, e.g.:

 - Lost connection to server
 - Timeout
 - could not open connection 'default' (on test startup)
 - failed to start mysqld
 - mysqltest returned unexpected code X, it has probably crashed

In all these cases, it's likely that the server is still running, but mtr currently
doesn't print anything that can be used to debug the failure (except SHOW PROCESSLIST for
timeout, as of recently).

To make these situations debuggable, mtr should after *every* failure iterate through all
started servers and:

 (1) check if mysqld is still running. If it is, get stack traces for every thread. Then
kill the server so that a core is saved.

 (2) check if a core is saved. If it is, and no stack trace was printed in the previous
step, get stack traces for every thread.

There are a lot of failures that can't be debugged currently, which would become
debuggable if we implemented this. Moreover, many of the failures that are debuggable
would be faster to debug if this was implemented.

How to repeat:
See, e.g., the following bugs: BUG#40142, BUG#38176, BUG#39570, BUG#38076, BUG#38096,
BUG#15399, BUG#39861, BUG#37714, BUG#38172, BUG#38288, BUG#38287, BUG#38289, BUG#38228,
BUG#38286. Most failures don't get reported though, since there is little point in
reporting something that can't be debugged...

Actually, I don't think *every* failure should generate a stack trace. Only failures where
the current server state may still be relevant should. That excludes result mismatches,
but includes all cases of timeout, wrong error code from server, or lost connection.

A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/63665

2714 Bjorn Munch	2009-01-21
      Bug #40399 Please make mtr print stack trace after every failure
      SIGABRT is sent to relevant processes after a timeout

A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/63688

2787 He Zhenxing	2009-01-21 [merge]
      Auto merge

A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/63689

2716 He Zhenxing	2009-01-21 [merge]
      Auto merge

Pushed into 6.0.10-alpha (revid:luis.soares@sun.com-20090129165607-wiskabxm948yx463)
(version source revid:luis.soares@sun.com-20090129163120-e2ntks4wgpqde6zt) (merge vers:
6.0.10-alpha) (pib:6)

Pushed into 5.1.32 (revid:luis.soares@sun.com-20090129165946-d6jnnfqfokuzr09y) (version
source revid:msvensson@mysql.com-20090121134150-3trohhf2oc9bwhxr) (merge vers: 5.1.31)
(pib:6)

Test suite changes. No changelog entry needed.

Pushed into 5.1.32-ndb-6.3.23 (revid:tomas.ulin@sun.com-20090217131017-6u8qz1edkjfiobef)
(version source revid:tomas.ulin@sun.com-20090203133556-9rclp06ol19bmzs4) (merge vers:
5.1.32-ndb-6.3.22) (pib:6)

Pushed into 5.1.32-ndb-6.4.3 (revid:tomas.ulin@sun.com-20090217134419-5ha6xg4dpedrbmau)
(version source revid:tomas.ulin@sun.com-20090203133556-9rclp06ol19bmzs4) (merge vers:
5.1.32-ndb-6.3.22) (pib:6)

Pushed into 5.1.32-ndb-6.2.17 (revid:tomas.ulin@sun.com-20090217134216-5699eq74ws4oxa0j)
(version source revid:tomas.ulin@sun.com-20090201210519-vehobc4sy3g9s38e) (merge vers:
5.1.32-ndb-6.2.17) (pib:6)