Bug #83456 Race condition in mysql_set_character_set, ASAN failure with multi-source rpl
Submitted: 20 Oct 2016 10:38 Modified: 24 May 2017 9:54
Reporter: Sven Sandberg Email Updates:
Status: Closed Impact on me:
None 
Category:MySQL Server: Charsets Severity:S3 (Non-critical)
Version:5.7 OS:Any
Assigned to: CPU Architecture:Any

[20 Oct 2016 10:38] Sven Sandberg 
Description:
mysql_set_character_set and mysql_set_character_set_with_default_collation (both defined in client.cc) will temporarily change the global variable charsets_dir. This makes them non-thread-safe.

In particular, this can cause access to freed memory in the replication receiver thread (aka IO thread) when using multi-source replication. In multi-source replication, two such threads are likely to start executing at nearly the same time. Then both threads can do the following at about the same time:
 1. From connect_to_master:
     1.1. call mysql_options(mysql, MYSQL_SET_CHARSET_DIR, (char *) charsets_dir), which will
     1.2. set mysql->options.charset_dir to strdup(charsets_dir).
 2. From connect_to_master:
     2.1. call mysql_reconnect, which will call
     2.2. mysql_set_character_set, which will
     2.3. temporarily change the global charsets_dir to mysql->options.charset_dir, then
     2.4. do some work, and finally
     2.5. change back charsets_dir to its previous value
 3. From handle_slave_io, call mysql_close, which calls mysql_close_free_options, which frees mysql->options.charset_dir.

Since no locks are protecting the global charsets_dir, the following can happen:

 1. T1 executes 1, 2.1, 2.2, 2.3, leaving the global charsets_dir set to the T1's mysql->options.charset_dir.
 2. T2 executes 1.1 (thus passing T1's mysql->options.charset_dir as the third argument to mysql_set_options)
 3. T1 continues to execute 2.4, 2.5, 3 (restoring the global charsets_dir and freeing T1's mysql->options.charset_dir)
 4. T2 executes 2.2, where strdup will read freed memory.

How to repeat:
This would explain the failure of rpl.rpl_multi_source_slave_files at http://pb2.no.oracle.com/?action=archive_download&archive_id=20851995&pretty=please

(The reason that the memory access in AddressSanitizer output is 32 bytes into the freed memory block is that my_malloc (in my_malloc.c) adds a 32 byte header.)

Suggested fix:
Don't modify charsets_dir. Instead, pass the string you need as parameter to whatever function it is used in.
[20 Oct 2016 10:41] Sven Sandberg 
Posted by developer:
 
Note: the non-thread-safeness has probably been there since forever, but it probably only manifests itself in 5.7 because we introduced multi-source replication which will call these functions from different threads at about the same time. I don't think the server would use different MYSQL objects concurrently before that. Therefore I set the version to 5.7.
[20 Oct 2016 12:06] Sven Sandberg 
Posted by developer:
 
Last line of Description should be:
 4. T2 executes 1.2, where strdup will read freed memory.
(Thanks Daogang for spotting that.)
[24 May 2017 9:54] Erlend Dahl 
Fixed in 8.0.2.