Bug #73714 innodb.innodb_bug-13628249 server shutdown hang
Submitted: 25 Aug 2014 18:37 Modified: 1 Jul 2016 16:57
Reporter: Mattias Jonsson Email Updates:
Status: Closed Impact on me:
None 
Category:MySQL Server: InnoDB storage engine Severity:S3 (Non-critical)
Version:5.7 OS:Any
Assigned to: CPU Architecture:Any

[25 Aug 2014 18:37] Mattias Jonsson 
Description:
innodb.innodb_bug-13628249 test fails during server shutdown due to hang.

See:
http://pb2.no.oracle.com/?template=mysql_show_test_failure&test_failure_id=5573526
http://pb2.no.oracle.com/?template=mysql_show_test_failure&test_failure_id=5573544

I suspect the thread running innobase_end :

 	mysqld-debug.exe!os_thread_sleep(unsigned __int64 tm) Line 274	C++
>	mysqld-debug.exe!logs_empty_and_mark_files_at_shutdown() Line 2018	C++
 	mysqld-debug.exe!innobase_shutdown_for_mysql() Line 2583	C++
 	mysqld-debug.exe!innobase_end(handlerton * hton, ha_panic_function type) Line 3446	C++
 	mysqld-debug.exe!ha_finalize_handlerton(st_plugin_int * plugin) Line 618	C++
 	mysqld-debug.exe!plugin_deinitialize(st_plugin_int * plugin, bool ref_check) Line 950	C++
 	mysqld-debug.exe!reap_plugins() Line 1029	C++
 	mysqld-debug.exe!plugin_shutdown() Line 1745	C++
 	mysqld-debug.exe!clean_up(bool print_message) Line 1372	C++
 	mysqld-debug.exe!win_main(int argc, char * * argv) Line 4799	C++
 	mysqld-debug.exe!mysql_service(void * p) Line 4819	C++
 	mysqld-debug.exe!mysqld_main(int argc, char * * argv) Line 5016	C++
 	mysqld-debug.exe!main(int argc, char * * argv) Line 26	C++
 	[External Code]	

srv_shutdown_state = SRV_SHUTDOWN_NONE
count = 597 in mysqld.4988.dmp and 503 in mysqld.4332.dmp

How to repeat:
See PB2, not sure what triggers it.
[25 Aug 2014 18:41] Mattias Jonsson 
Parallel Stacks from Visual Studio

Attachment: innodb_bug-13628249.png (image/png, text), 26.31 KiB.
[13 Apr 2015 11:36] Marko Mäkelä 
Posted by developer:
 
In the test innodb.innodb_bug-13628249, the hang is caused, because with innodb_force_recovery=3 (and larger) the rollback of recovered transactions is disabled.
I checked one core dump.
It had one active transactions that had inserted the 4 records (1),(2),(3),(4) to the table t1.
It does not make sense to wait for the rollback to finish (or this recovered incomplete transaction to disappear) at shutdown, if the rollback has been disabled by the startup parameter.

This test is not failing every time, because the kill point with regard to the redo log LSN is nondeterministic:

CREATE TABLE t1(c1 INT PRIMARY KEY) ENGINE=InnoDB STATS_PERSISTENT=0;
BEGIN;
INSERT INTO t1 VALUES(1), (2), (3), (4);

--echo # Restart the server in force recovery mode (3)
let $restart_parameters = restart: --innodb-force-recovery=3;
--source include/kill_and_restart_mysqld.inc

The test only fails if the redo log for the INSERT actually was written to the redo log file.
If we want to make this fail every time, we should create another connection and commit another transaction there. Then, that transaction commit would force the redo log for the above INSERT statement to be recovered, and the subsequent shutdown should hang every time.
[1 Jul 2016 16:57] Paul DuBois 
Posted by developer:
 
Fixed in 8.0.0.

Work was done for test suite. No changelog entry needed.