MySQL Bugs: #67156: Sporadic query cache related crash in pthread_rwlock

Bug #67156	Sporadic query cache related crash in pthread_rwlock_init()
Submitted:	9 Oct 2012 18:21	Modified:	10 Dec 2012 17:02
Reporter:	Davi Arnaut (OCA)	Email Updates:
Status:	Closed	Impact on me:	None
Category:	MySQL Server: Query Cache	Severity:	S2 (Serious)
Version:	5.5.27	OS:	MacOS (10.7.5)
Assigned to:		CPU Architecture:	Any
Tags:	crash, Mac OS X, pthread_rwlock_init, query_cache

Description:
Reinitializing the query cache might lead to a crash inside pthread_rwlock_init() on Mac OS X. The problem is twofold:

First, Mac OS X's pthread_rwlock_init() implements a check to detect attempts to reinitialize a previously initialized but not yet destroyed read/write lock. This check works by looking for a signature within the rwlock object and, if the signature matches, some pointer variables within the object are dereferenced.

Second, when MySQL's query cache is resized (e.g. SET query_cache_size = ...), or simply deinitialized, the memory used to allocate blocks is freed without explicitly deinitializing the allocated blocks. In particular, 'queries blocks' contain rwlock objects that are not properly destroyed.

Hence, If the query cache is resized, and the same memory region is used, and a new query block is allocated in the same position, attempting to initialize the block's read/write lock might crash if the read-write lock object signature was preserved but yet somehow some of its fields were corrupted. In the observed crashes, the signature remains intact but the pointer variables that are dereferenced were set to null.

How to repeat:
 ./mtr --repeat=1000 --ps-protocol --mysqld=--binlog-format=row query_cache_type_func

See pthread_rwlock_init in http://www.opensource.apple.com/source/Libc/Libc-763.13/pthreads/pthread_rwlock.c for reference.

Backtrace:

#0  0x00007fff8aa5c2a2 in pthread_rwlock_init ()
#1  0x000000010b9630b4 in inline_mysql_rwlock_init (that=0x7fe301881938) at mysql_thread.h:691
#2  0x000000010b96e870 in Query_cache_query::init_n_lock (this=0x7fe301881930) at sql_cache.cc:788
#3  0x000000010b96cb4a in Query_cache::store_query (this=0x10c3d3760, thd=0x7fe301873800, tables_used=0x7fe301876810) at sql_cache.cc:1316
#4  0x000000010b9d5719 in execute_sqlcom_select (thd=0x7fe301873800, all_tables=0x7fe301876810) at sql_parse.cc:4715
#5  0x000000010b9d670a in mysql_execute_command (thd=0x7fe301873800) at sql_parse.cc:2273
#6  0x000000010ba12d99 in Prepared_statement::execute (this=0x7fe300c020f0, expanded_query=0x10f396a78, open_cursor=false) at sql_prepare.cc:3799
#7  0x000000010ba13311 in Prepared_statement::execute_loop (this=0x7fe300c020f0, expanded_query=0x10f396a78, open_cursor=false, packet=0x7fe30187cc0a "", packet_end=0x7fe30187cc0a "") at sql_prepare.cc:3458
#8  0x000000010ba13ac3 in mysqld_stmt_execute (thd=0x7fe301873800, packet_arg=0x7fe30187cc01 "\f", packet_length=9) at sql_prepare.cc:2579
#9  0x000000010b9e28f0 in dispatch_command (command=COM_STMT_EXECUTE, thd=0x7fe301873800, packet=0x7fe30187cc01 "\f", packet_length=9) at sql_parse.cc:1098
#10 0x000000010b9e4a01 in do_command (thd=0x7fe301873800) at sql_parse.cc:775
#11 0x000000010bb36713 in do_handle_one_connection (thd_arg=0x7fe301873800) at sql_connect.cc:840
#12 0x000000010bb36800 in handle_one_connection (arg=0x7fe301873800) at sql_connect.cc:759
#13 0x00007fff8aa1f8bf in _pthread_start ()
#14 0x00007fff8aa22b75 in thread_start ()

Properly destroy read-write lock objects used by query cache blocks. (*) I confirm the code being submitted is offered under the terms of the OCA, and that I am authorized to contribute it.

Contribution: Bug#67156-Sporadic-query-cache-related-crash-in-pthr.patch (application/octet-stream, text), 2.05 KiB.

Hi Davi!

Verified using 5.5.29 on OSX 10.8.2.
Still trying to reproduce on 5.6.8.

Also reproduced on 5.6.8 (unsurprisingly!)

Jon, can you please check if this might be causing a problem I reported:
Bug 14727815 - WINDOWS - CRASH IN SRW_UNLOCK IN QUERY CACHE CODE

Noted in 5.6.9, 5.7.0 changelogs.

On Mac OS X, reinitializing the query cache could cause the server to
exit. Thanks to Davi Arnaut for the patch.