MySQL Bugs: #40399: Please make mtr print stack trace after every failure

Bug #40399	Please make mtr print stack trace after every failure
Submitted:	29 Oct 2008 17:29	Modified:	30 Jan 2009 17:46
Reporter:	Sven Sandberg	Email Updates:
Status:	Closed	Impact on me:	None
Category:	Tests	Severity:	S7 (Test Cases)
Version:	5.1	OS:	Any
Assigned to:	Bjørn Munch	CPU Architecture:	Any
Tags:	mtr

Description:
There are a number of common test failures that cannot be debugged. The error messages can be, e.g.:

 - Lost connection to server
 - Timeout
 - could not open connection 'default' (on test startup)
 - failed to start mysqld
 - mysqltest returned unexpected code X, it has probably crashed

In all these cases, it's likely that the server is still running, but mtr currently doesn't print anything that can be used to debug the failure (except SHOW PROCESSLIST for timeout, as of recently).

To make these situations debuggable, mtr should after *every* failure iterate through all started servers and:

 (1) check if mysqld is still running. If it is, get stack traces for every thread. Then kill the server so that a core is saved.

 (2) check if a core is saved. If it is, and no stack trace was printed in the previous step, get stack traces for every thread.

There are a lot of failures that can't be debugged currently, which would become debuggable if we implemented this. Moreover, many of the failures that are debuggable would be faster to debug if this was implemented.

How to repeat:
See, e.g., the following bugs: BUG#40142, BUG#38176, BUG#39570, BUG#38076, BUG#38096, BUG#15399, BUG#39861, BUG#37714, BUG#38172, BUG#38288, BUG#38287, BUG#38289, BUG#38228, BUG#38286. Most failures don't get reported though, since there is little point in reporting something that can't be debugged...

Actually, I don't think *every* failure should generate a stack trace. Only failures where the current server state may still be relevant should. That excludes result mismatches, but includes all cases of timeout, wrong error code from server, or lost connection.

A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/63665

2714 Bjorn Munch	2009-01-21
      Bug #40399 Please make mtr print stack trace after every failure
      SIGABRT is sent to relevant processes after a timeout

A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/63688

2787 He Zhenxing	2009-01-21 [merge]
      Auto merge

A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/63689

2716 He Zhenxing	2009-01-21 [merge]
      Auto merge

Pushed into 6.0.10-alpha (revid:luis.soares@sun.com-20090129165607-wiskabxm948yx463) (version source revid:luis.soares@sun.com-20090129163120-e2ntks4wgpqde6zt) (merge vers: 6.0.10-alpha) (pib:6)

Pushed into 5.1.32 (revid:luis.soares@sun.com-20090129165946-d6jnnfqfokuzr09y) (version source revid:msvensson@mysql.com-20090121134150-3trohhf2oc9bwhxr) (merge vers: 5.1.31) (pib:6)

Test suite changes. No changelog entry needed.

Pushed into 5.1.32-ndb-6.3.23 (revid:tomas.ulin@sun.com-20090217131017-6u8qz1edkjfiobef) (version source revid:tomas.ulin@sun.com-20090203133556-9rclp06ol19bmzs4) (merge vers: 5.1.32-ndb-6.3.22) (pib:6)

Pushed into 5.1.32-ndb-6.4.3 (revid:tomas.ulin@sun.com-20090217134419-5ha6xg4dpedrbmau) (version source revid:tomas.ulin@sun.com-20090203133556-9rclp06ol19bmzs4) (merge vers: 5.1.32-ndb-6.3.22) (pib:6)

Pushed into 5.1.32-ndb-6.2.17 (revid:tomas.ulin@sun.com-20090217134216-5699eq74ws4oxa0j) (version source revid:tomas.ulin@sun.com-20090201210519-vehobc4sy3g9s38e) (merge vers: 5.1.32-ndb-6.2.17) (pib:6)