MySQL Bugs: #72809: InnoDB Linux native aio setup missing barrier after setup

Bug #72809	InnoDB Linux native aio setup missing barrier after setup
Submitted:	30 May 2014 7:20	Modified:	16 Sep 2014 23:15
Reporter:	Stewart Smith	Email Updates:
Status:	Closed	Impact on me:	None
Category:	MySQL Server: InnoDB storage engine	Severity:	S2 (Serious)
Version:	5.6, 5.7	OS:	Any
Assigned to:		CPU Architecture:	Any
Tags:	MySQL, PowerPC

Description:
I've observed this on 5.6.17, and likely too on 5.7 (but I have other issues with my 5.7 current patchset).

Basically, as InnoDB sets up the data structures for aio during startup, it creates a mutex but doesn't take it, thus not issuing any memory barriers.

It turns out that a POWER8 seems to be fast enough and with enough caches that you can get into runtime inside InnoDB with InnoDB doing IO where one of the InnoDB IO threads will hit an assert due to reading stale data left over from setting up AIO rather than runtime.

A simple __sync_synchronize(); at the end of setting up the AIO data structures seems to fix it.

How to repeat:
I've reproduced on POWER8 rather reliably, with the problem going away with my patch. We didn't really expect this to happen, but the explanation seems reasonable and the patch does fix it... I guess if I was more patient between server startup and loading data for benchmark I wouldn't see it and the CPU would manage to get the cachelines out.

Suggested fix:
Index: mysql-5.7.4-m14/storage/innobase/os/os0file.cc
===================================================================
--- mysql-5.7.4-m14.orig/storage/innobase/os/os0file.cc
+++ mysql-5.7.4-m14/storage/innobase/os/os0file.cc
@@ -3741,7 +3741,7 @@ skip_native_aio:
 		slot->ret = 0;
 #endif /* WIN_ASYNC_IO */
 	}
-
+	__sync_synchronize();
 	return(array);
 }

Memory barrier after InnoDB native Linux AIO setup (*) I confirm the code being submitted is offered under the terms of the OCA, and that I am authorized to contribute it.

Contribution: innodb-aio-barriers.patch (text/x-patch), 365 bytes.

Hello Stewart,

Thank you for the report and contribution.

Thanks,
Umesh

Fixed as of the upcoming 5.6.22 and 5.7.6 releases, and here's the changelog entry:

For explicit cache coherency, a write barrier was added to the head of
"os_thread_create_func()", and a read barrier was added to before the
assertion code in "rw_lock_free_func()". 

This bug fix also address Bug #62692.

Thank you for the bug report.

$ bzr log -r 6163
------------------------------------------------------------
revno: 6163
committer: Yasufumi Kinoshita <yasufumi.kinoshita@oracle.com>
branch nick: mysql-5.6
timestamp: Tue 2014-09-16 16:46:24 +0900
message:
  Some memory barrier were added for non-intel architecture (POWER, ARM, etc..)
  
  Bug#18870970 : INNODB LINUX NATIVE AIO SETUP MISSING BARRIER AFTER SETUP (Bug#72809)
  For explicit cache coherency with the parent thread, write barrier is added to the head of os_thread_create_func()
  
  Bug#13364876 : RW_LOCK_FREE MIGHT CAUSE ASSERTION FAILURE BY RACE CONDITION (Bug#62692)
  For explicit cache coherency with the other thread, read barrier is added to before the assertion codes at rw_lock_free_func()
  
  Approved by Vasil in rb#6255