Bug #67156 Sporadic query cache related crash in pthread_rwlock_init()
Submitted: 9 Oct 2012 18:21 Modified: 10 Dec 2012 17:02
Reporter: Davi Arnaut (OCA) Email Updates:
Status: Closed Impact on me:
None 
Category:MySQL Server: Query Cache Severity:S2 (Serious)
Version:5.5.27 OS:MacOS (10.7.5)
Assigned to: CPU Architecture:Any
Tags: crash, Mac OS X, pthread_rwlock_init, query_cache

[9 Oct 2012 18:21] Davi Arnaut
Description:
Reinitializing the query cache might lead to a crash inside pthread_rwlock_init() on Mac OS X. The problem is twofold:

First, Mac OS X's pthread_rwlock_init() implements a check to detect attempts to reinitialize a previously initialized but not yet destroyed read/write lock. This check works by looking for a signature within the rwlock object and, if the signature matches, some pointer variables within the object are dereferenced.

Second, when MySQL's query cache is resized (e.g. SET query_cache_size = ...), or simply deinitialized, the memory used to allocate blocks is freed without explicitly deinitializing the allocated blocks. In particular, 'queries blocks' contain rwlock objects that are not properly destroyed.

Hence, If the query cache is resized, and the same memory region is used, and a new query block is allocated in the same position, attempting to initialize the block's read/write lock might crash if the read-write lock object signature was preserved but yet somehow some of its fields were corrupted. In the observed crashes, the signature remains intact but the pointer variables that are dereferenced were set to null.

How to repeat:
 ./mtr --repeat=1000 --ps-protocol --mysqld=--binlog-format=row query_cache_type_func
[9 Oct 2012 18:21] Davi Arnaut
See pthread_rwlock_init in http://www.opensource.apple.com/source/Libc/Libc-763.13/pthreads/pthread_rwlock.c for reference.
[9 Oct 2012 18:25] Davi Arnaut
Backtrace:

#0  0x00007fff8aa5c2a2 in pthread_rwlock_init ()
#1  0x000000010b9630b4 in inline_mysql_rwlock_init (that=0x7fe301881938) at mysql_thread.h:691
#2  0x000000010b96e870 in Query_cache_query::init_n_lock (this=0x7fe301881930) at sql_cache.cc:788
#3  0x000000010b96cb4a in Query_cache::store_query (this=0x10c3d3760, thd=0x7fe301873800, tables_used=0x7fe301876810) at sql_cache.cc:1316
#4  0x000000010b9d5719 in execute_sqlcom_select (thd=0x7fe301873800, all_tables=0x7fe301876810) at sql_parse.cc:4715
#5  0x000000010b9d670a in mysql_execute_command (thd=0x7fe301873800) at sql_parse.cc:2273
#6  0x000000010ba12d99 in Prepared_statement::execute (this=0x7fe300c020f0, expanded_query=0x10f396a78, open_cursor=false) at sql_prepare.cc:3799
#7  0x000000010ba13311 in Prepared_statement::execute_loop (this=0x7fe300c020f0, expanded_query=0x10f396a78, open_cursor=false, packet=0x7fe30187cc0a "", packet_end=0x7fe30187cc0a "") at sql_prepare.cc:3458
#8  0x000000010ba13ac3 in mysqld_stmt_execute (thd=0x7fe301873800, packet_arg=0x7fe30187cc01 "\f", packet_length=9) at sql_prepare.cc:2579
#9  0x000000010b9e28f0 in dispatch_command (command=COM_STMT_EXECUTE, thd=0x7fe301873800, packet=0x7fe30187cc01 "\f", packet_length=9) at sql_parse.cc:1098
#10 0x000000010b9e4a01 in do_command (thd=0x7fe301873800) at sql_parse.cc:775
#11 0x000000010bb36713 in do_handle_one_connection (thd_arg=0x7fe301873800) at sql_connect.cc:840
#12 0x000000010bb36800 in handle_one_connection (arg=0x7fe301873800) at sql_connect.cc:759
#13 0x00007fff8aa1f8bf in _pthread_start ()
#14 0x00007fff8aa22b75 in thread_start ()
[9 Oct 2012 18:40] Davi Arnaut
Properly destroy read-write lock objects used by query cache blocks.

(*) I confirm the code being submitted is offered under the terms of the OCA, and that I am authorized to contribute it.

Contribution: Bug#67156-Sporadic-query-cache-related-crash-in-pthr.patch (application/octet-stream, text), 2.05 KiB.

[10 Oct 2012 7:22] Jon Olav Hauglid
Hi Davi!

Verified using 5.5.29 on OSX 10.8.2.
Still trying to reproduce on 5.6.8.
[10 Oct 2012 8:39] Jon Olav Hauglid
Also reproduced on 5.6.8 (unsurprisingly!)
[11 Oct 2012 7:21] MySQL Verification Team
Jon, can you please check if this might be causing a problem I reported:
Bug 14727815 - WINDOWS - CRASH IN SRW_UNLOCK IN QUERY CACHE CODE
[10 Dec 2012 17:02] Paul DuBois
Noted in 5.6.9, 5.7.0 changelogs.

On Mac OS X, reinitializing the query cache could cause the server to
exit. Thanks to Davi Arnaut for the patch.