Bug #72809 InnoDB Linux native aio setup missing barrier after setup
Submitted: 30 May 2014 7:20 Modified: 16 Sep 2014 23:15
Reporter: Stewart Smith (OCA) Email Updates:
Status: Closed Impact on me:
None 
Category:MySQL Server: InnoDB storage engine Severity:S2 (Serious)
Version:5.6, 5.7 OS:Any
Assigned to:
Tags: MySQL, PowerPC

[30 May 2014 7:20] Stewart Smith
Description:
I've observed this on 5.6.17, and likely too on 5.7 (but I have other issues with my 5.7 current patchset).

Basically, as InnoDB sets up the data structures for aio during startup, it creates a mutex but doesn't take it, thus not issuing any memory barriers.

It turns out that a POWER8 seems to be fast enough and with enough caches that you can get into runtime inside InnoDB with InnoDB doing IO where one of the InnoDB IO threads will hit an assert due to reading stale data left over from setting up AIO rather than runtime.

A simple __sync_synchronize(); at the end of setting up the AIO data structures seems to fix it.

How to repeat:
I've reproduced on POWER8 rather reliably, with the problem going away with my patch. We didn't really expect this to happen, but the explanation seems reasonable and the patch does fix it... I guess if I was more patient between server startup and loading data for benchmark I wouldn't see it and the CPU would manage to get the cachelines out.

Suggested fix:
Index: mysql-5.7.4-m14/storage/innobase/os/os0file.cc
===================================================================
--- mysql-5.7.4-m14.orig/storage/innobase/os/os0file.cc
+++ mysql-5.7.4-m14/storage/innobase/os/os0file.cc
@@ -3741,7 +3741,7 @@ skip_native_aio:
 		slot->ret = 0;
 #endif /* WIN_ASYNC_IO */
 	}
-
+	__sync_synchronize();
 	return(array);
 }
[30 May 2014 7:21] Stewart Smith
Memory barrier after InnoDB native Linux AIO setup

(*) I confirm the code being submitted is offered under the terms of the OCA, and that I am authorized to contribute it.

Contribution: innodb-aio-barriers.patch (text/x-patch), 365 bytes.

[30 May 2014 8:09] Umesh Shastry
Hello Stewart,

Thank you for the report and contribution.

Thanks,
Umesh
[16 Sep 2014 23:15] Daniel Price
Fixed as of the upcoming 5.6.22 and 5.7.6 releases, and here's the changelog entry:

For explicit cache coherency, a write barrier was added to the head of
"os_thread_create_func()", and a read barrier was added to before the
assertion code in "rw_lock_free_func()". 

This bug fix also address Bug #62692.

Thank you for the bug report.
[10 Dec 2014 13:03] Laurynas Biveinis
$ bzr log -r 6163
------------------------------------------------------------
revno: 6163
committer: Yasufumi Kinoshita <yasufumi.kinoshita@oracle.com>
branch nick: mysql-5.6
timestamp: Tue 2014-09-16 16:46:24 +0900
message:
  Some memory barrier were added for non-intel architecture (POWER, ARM, etc..)
  
  Bug#18870970 : INNODB LINUX NATIVE AIO SETUP MISSING BARRIER AFTER SETUP (Bug#72809)
  For explicit cache coherency with the parent thread, write barrier is added to the head of os_thread_create_func()
  
  Bug#13364876 : RW_LOCK_FREE MIGHT CAUSE ASSERTION FAILURE BY RACE CONDITION (Bug#62692)
  For explicit cache coherency with the other thread, read barrier is added to before the assertion codes at rw_lock_free_func()
  
  Approved by Vasil in rb#6255