Bug #70860 --tc-heuristic-recover option values are broken
Submitted: 8 Nov 2013 10:20 Modified: 11 Jun 2015 16:35
Reporter: Laurynas Biveinis (OCA) Email Updates:
Status: Closed Impact on me:
None 
Category:MySQL Server: Logging Severity:S3 (Non-critical)
Version:5.6,5.7 OS:Any
Assigned to: CPU Architecture:Any

[8 Nov 2013 10:20] Laurynas Biveinis
Description:
--tc-heuristic-recover option values are broken due to option parser enumerating them starting from 0 and the source code enumerating them starting from 1:

#define TC_HEURISTIC_RECOVER_COMMIT   1
#define TC_HEURISTIC_RECOVER_ROLLBACK 2

How to repeat:
Set up the server for 2 XA SEs as in bug 70854, then start the server under the debugger with --tc-heuristic-recover=ROLLBACK, set a breakpoint on ha_recover() (by the time of its call tc_heuristic_recover variable must be set up):

Breakpoint 2, ha_recover (commit_list=0x0) at /home/laurynas/percona/src/mysql-server/sql/handler.cc:1846
1846	  DBUG_ENTER("ha_recover");
(gdb) print tc_heuristic_recover
$1 = 1

that is it's, equal to TC_HEURISTIC_RECOVER_COMMIT instead.

Suggested fix:
Re-enumerating the #defines to start from 0 won't work, as zero value for tc_heuristic_recover should mean that the option was not specified at all.

I'd consider adding "NONE" as the 1st allowed option value and the default.
[9 Jul 2014 11:29] Laurynas Biveinis
5.6 is affected too (probably all the lower versions as well, haven't checked).

Here is a testcase, maybe it will help to verify this bug. With the bug present, the testcase will process --tc-heuristic-recover=ROLLBACK as COMMIT and COMMIT as option not specified (i.e. testcase will hang on that exec $MYSQLD_CMD call).

--source include/not_embedded.inc
--source include/not_valgrind.inc
--source include/have_debug.inc
--source include/have_innodb.inc
--source include/have_log_bin.inc

let $restart_file= $MYSQLTEST_VARDIR/tmp/mysqld.1.expect;
let $restart_log= $MYSQLTEST_VARDIR/log/mysqld.1.restart.err;

CREATE TABLE t1 (a INT) ENGINE=InnoDB;

INSERT INTO t1 VALUES (1);

BEGIN;
INSERT INTO t1 VALUES (2);
SET SESSION debug="d,crash_commit_after_prepare";
--exec echo "wait" > $restart_file
--error 2006, 2013
COMMIT;

--source include/wait_until_disconnected.inc
--echo Restarting with --tc-heuristic-recover=ROLLBACK
--error 1
--exec $MYSQLD_CMD --tc-heuristic-recover=ROLLBACK --console > $restart_log 2>&1
--exec echo "restart" > $restart_file
--enable_reconnect
--source include/wait_until_connected_again.inc

--echo 2 should be absent:
SELECT * FROM t1;

BEGIN;
INSERT INTO t1 VALUES (2);
SET SESSION debug="d,crash_commit_after_prepare";
--exec echo "wait" > $restart_file
--error 2006, 2013
COMMIT;

--source include/wait_until_disconnected.inc
--echo Restarting with --tc-heuristic-recover=COMMIT
--error 1
--exec $MYSQLD_CMD --tc-heuristic-recover=COMMIT --console >> $restart_log 2>&1
--exec echo "restart" > $restart_file
--enable_reconnect
--source include/wait_until_connected_again.inc

--echo 2 should be present:
SELECT * FROM t1;

DROP TABLE t1;
[9 Jul 2014 15:52] Laurynas Biveinis
Even though the testcase run on any server version shows that the option value is processed incorrectly, its actual outcome after fix differs on different servers.

On 5.7 with 2 XA-supporting SEs (e.g. patched for XA EXAMPLE as in bug 70854), the server refuses to ROLLBACK. COMMIT appears to work. Server then crashes with a debug assert (will report this separately).
On stock 5.7, COMMIT is reset to ROLLBACK after the option processing. ROLLBACK appears to work. The server crashes with the same assert later.
On 5.6, COMMIT is reset to ROLLBACK after the option processing. ROLLBACK appears to work.
[9 Jul 2014 16:00] Laurynas Biveinis
Patch for 5.6

(*) I confirm the code being submitted is offered under the terms of the OCA, and that I am authorized to contribute it.

Contribution: 70860-oracle-5.6.patch (application/octet-stream, text), 4.05 KiB.

[9 Jul 2014 16:05] Laurynas Biveinis
The uploaded patch fixes option handling for 5.6 and adds a testcase. There was no public MTR testcase for --tc-heuristic-recover at all.

It has two drawbacks:
1) Needs re-recording of mysqld--help-win.
2) Does not touch the unconditional reset of COMMIT to ROLLBACK, which assumes that server is only running with a single 2PC SE, which is wrong for e.g. TokuDB. 5.7 handling of COMMIT and ROLLBACK is also questionable but that's for another time.
[9 Jul 2014 16:19] Laurynas Biveinis
The 5.7 crash is bug 73246.
[11 Jun 2015 16:35] Paul DuBois
Noted in 5.7.8, 5.8.0 changelogs.

The server interpreted --tc-heuristic-recover option values
incorrectly due to an off-by-one error. Thanks to Laurynas Biveinis
for the patch.