| Bug #82220 | Read of out-of-scope (tempfile) in mysql_update() | ||
|---|---|---|---|
| Submitted: | 13 Jul 2016 20:29 | Modified: | 28 Jul 2016 15:17 |
| Reporter: | David Gow | Email Updates: | |
| Status: | Closed | Impact on me: | |
| Category: | MySQL Server: Compiling | Severity: | S3 (Non-critical) |
| Version: | 5.7.11/5.7 git, 5.7.13 | OS: | Linux |
| Assigned to: | CPU Architecture: | Any | |
| Tags: | asan | ||
[20 Jul 2016 12:04]
MySQL Verification Team
Hello David Gow, Thank you for the report and feedback. Could you please share exact build details(cmake), and clang version used in your environment? I checked with 5.7.13 source build but not seeing any issues at my end. ushastry@ubuntu1604lts:~/Downloads/mysql-5.7.13$ export CC=/home/ushastry/Downloads/clang/bin/clang ushastry@ubuntu1604lts:~/Downloads/mysql-5.7.13$ export CXX=/home/ushastry/Downloads/clang/bin/clang++ ushastry@ubuntu1604lts:~/Downloads/mysql-5.7.13$ export LD=$CXX ushastry@ubuntu1604lts:~/Downloads/mysql-5.7.13$ export ASAN_SYMBOLIZER_PATH=/home/ushastry/Downloads/clang/bin/llvm-symbolizer ushastry@ubuntu1604lts:~/Downloads/mysql-5.7.13$ cmake . -DWITH_ASAN=ON -DCMAKE_BUILD_TYPE=Debug -DWITH_BOOST=./boost/ -- Running cmake version 3.5.1 -- Could NOT find Git (missing: GIT_EXECUTABLE) -- Configuring with MAX_INDEXES = 64U -- The C compiler identification is Clang 3.8.0 -- The CXX compiler identification is Clang 3.8.0 . . -- Library mysqlserver depends on OSLIBS -lpthread;m;rt;crypt;dl -- INSTALL mysqlclient.pc lib/pkgconfig -- CMAKE_BUILD_TYPE: Debug -- COMPILE_DEFINITIONS: _GNU_SOURCE;_FILE_OFFSET_BITS=64;HAVE_CONFIG_H -- CMAKE_C_FLAGS: -Wall -Wextra -Wformat-security -Wvla -Wwrite-strings -Wdeclaration-after-statement -- CMAKE_CXX_FLAGS: -Wall -Wextra -Wformat-security -Wvla -Woverloaded-virtual -Wno-unused-parameter -Wno-null-conversion -Wno-unused-private-field -- CMAKE_C_FLAGS_DEBUG: -g -fno-omit-frame-pointer -fno-strict-aliasing -fsanitize=address -O1 -Wno-error -fPIC -DENABLED_DEBUG_SYNC -DSAFE_MUTEX -- CMAKE_CXX_FLAGS_DEBUG: -g -fno-omit-frame-pointer -fno-strict-aliasing -fsanitize=address -O1 -Wno-error -fPIC -DENABLED_DEBUG_SYNC -DSAFE_MUTEX -- Configuring done -- Generating done -- Build files have been written to: /home/ushastry/Downloads/mysql-5.7.13 ushastry@ubuntu1604lts:~/Downloads/mysql-5.7.13$ make -j8 Scanning dependencies of target INFO_SRC Scanning dependencies of target INFO_BIN Scanning dependencies of target zlib Scanning dependencies of target taocrypt Scanning dependencies of target yassl Scanning dependencies of target protoclib . [100%] Built target mysqld Scanning dependencies of target udf_example [100%] Building CXX object sql/CMakeFiles/udf_example.dir/udf_example.cc.o [100%] Linking CXX shared module udf_example.so [100%] Built target udf_example ushastry@ubuntu1604lts:~/Downloads/mysql-5.7.13$ cd mysql-test/ ushastry@ubuntu1604lts:~/Downloads/mysql-5.7.13/mysql-test$ ./mtr main.update Logging: ./mtr main.update MySQL Version 5.7.13 Checking supported features... - SSL connections supported - binaries are debug compiled Collecting tests... Checking leftover processes... Removing old var directory... Creating var directory '/home/ushastry/Downloads/mysql-5.7.13/mysql-test/var'... Installing system database... ============================================================================== TEST RESULT TIME (ms) or COMMENT -------------------------------------------------------------------------- worker[1] Using MTR_BUILD_THREAD 300, with reserved ports 13000..13009 main.update [ pass ] 2164 -------------------------------------------------------------------------- The servers were restarted 0 times Spent 2.164 of 19 seconds executing testcases Completed: All 1 tests were successful. ushastry@ubuntu1604lts:~/Downloads/mysql-5.7.13/mysql-test$ cat /etc/*release DISTRIB_ID=Ubuntu DISTRIB_RELEASE=16.04 DISTRIB_CODENAME=xenial DISTRIB_DESCRIPTION="Ubuntu 16.04 LTS" NAME="Ubuntu" VERSION="16.04 LTS (Xenial Xerus)" ID=ubuntu ID_LIKE=debian PRETTY_NAME="Ubuntu 16.04 LTS" VERSION_ID="16.04" HOME_URL="http://www.ubuntu.com/" SUPPORT_URL="http://help.ubuntu.com/" BUG_REPORT_URL="http://bugs.launchpad.net/ubuntu/" UBUNTU_CODENAME=xenial Thanks, Umesh
[21 Jul 2016 23:18]
David Gow
This only shows up with the --fsanitize-address-use-after-scope option, which is not yet available in any released version of clang (and which is not set by default yet, so the CMake files need modification, too.) You ought to be able to reproduce it with an SVN build of clang/llvm, and the addition of that flag to the address sanitizer options. The problem is quite easy to see by inspecting the code, though — the assignment here: https://github.com/mysql/mysql-server/blob/5.7/sql/sql_update.cc#L740 which copies the entire IO_CACHE structure. This means that the current_* pointers: https://github.com/mysql/mysql-server/blob/5.7/include/my_sys.h#L389 will still be pointing within the original tempfile variable, which will go out-of-scope. — David
[22 Jul 2016 22:44]
David Gow
There was a regression in llvm/clang the last few days which broke building mysql completely. It's fixed now, but just in case: here are some steps for building a version with this test. You'll probably want to look at the clang page <http://clang.llvm.org/get_started.html> for general guidance, but I've included the exact SVN revisions which worked. It's also worth noting that, on Ubuntu 14.04, CMake was too old to build llvm/clang, so I compiled and used CMake 3.6.0. $ mkdir llvm-build $ cd llvm-build $ svn co -r 276452 http://llvm.org/svn/llvm-project/llvm/trunk llvm $ cd llvm/tools $ svn co -r 276448 http://llvm.org/svn/llvm-project/cfe/trunk clang $ cd clang/tools $ svn co -r 276282 http://llvm.org/svn/llvm-project/clang-tools-extra/trunk extra $ cd ../../../projects $ svn co -r 276299 http://llvm.org/svn/llvm-project/compiler-rt/trunk compiler-rt $ cd ../.. $ mkdir build $ cd build # You'll need a pretty recent CMake for this — I used 3.6.0 $ cmake ../llvm $ make Then, I modified mysql's CMakeLists.txt to update the WITH_ASAN call to MY_SANITIZER_CHECK to read: MY_SANITIZER_CHECK("-fsanitize=address -fsanitize-address-use-after-scope" WITH_ASAN_OK) and built with: $ cmake .. -DCMAKE_BUILD_TYPE=Debug -DWITH_ASAN=1 -DCMAKE_C_COMPILER=${PATH_TO_LLVM}/build/bin/clang -DCMAKE_CXX_COMPILER=${PATH_TO_LLVM}/build/bin/clang++ -DDOWNLOAD_BOOST=1 -DWITH_BOOST=boost $ make And reproduced the issue with $ cd mysql-test $ ./mtr main.update This successfully reproduced the issue.
[24 Jul 2016 5:27]
MySQL Verification Team
Thank you David, verified as described. Thanks, Umesh
[24 Jul 2016 5:29]
MySQL Verification Team
Build log and test results
Attachment: 82220.build (application/octet-stream, text), 733.66 KiB.
[28 Jul 2016 15:17]
Paul DuBois
Posted by developer: Fixed in 8.0.0. Code cleanup. No changelog entry needed.
[4 Nov 2017 9:42]
MySQL Verification Team
I know this is fixed in 8.0+ but thought I write the 5.7 output here.
Testcase
---------
set sql_mode="";
drop table if exists t;
create table t(a tinyint primary key) engine=innodb;
update t set a='a';
----------------------
Version: '5.7.21-asan' socket: '/tmp/mysql.sock' port: 3306 (Built on 04 November 2017 with gcc (GCC) 8.0.0 20170924 (experimental))
2017-11-04T09:35:34.026706Z 0 [Note] Executing 'SELECT * FROM INFORMATION_SCHEMA.TABLES;' to get a list of tables using the deprecated partition engine. You may use the startup option '--disable-partition-engine-check' to skip this check.
2017-11-04T09:35:34.027165Z 0 [Note] Beginning of list of non-natively partitioned tables
2017-11-04T09:35:34.234160Z 0 [Note] End of list of non-natively partitioned tables
2017-11-04T09:35:35.117727Z 3 [Warning] IP address '192.168.1.107' could not be resolved: Name or service not known
=================================================================
==3829==ERROR: AddressSanitizer: stack-use-after-scope on address 0x7f295417cae0 at pc 0x0000024235ad bp 0x7f295417bfd0 sp 0x7f295417bfc8
READ of size 8 at 0x7f295417cae0 thread T30
#0 0x24235ac in reinit_io_cache /home/sbester/git/mysql-git/mysys/mf_iocache.c:328
#1 0x1ab714a in init_read_record(READ_RECORD*, THD*, TABLE*, QEP_TAB*, int, bool, bool) /home/sbester/git/mysql-git/sql/records.cc:248
#2 0x1eab9e8 in mysql_update(THD*, List<Item>&, List<Item>&, unsigned long long, enum_duplicates, unsigned long long*, unsigned long long*) /home/sbester/git/mysql-git/sql/sql_update.cc:765
#3 0x1eb5f11 in Sql_cmd_update::try_single_table_update(THD*, bool*) /home/sbester/git/mysql-git/sql/sql_update.cc:2882
#4 0x1eb6a78 in Sql_cmd_update::execute(THD*) /home/sbester/git/mysql-git/sql/sql_update.cc:3009
#5 0x1cd4c8d in mysql_execute_command(THD*, bool) /home/sbester/git/mysql-git/sql/sql_parse.cc:3576
#6 0x1ce039c in mysql_parse(THD*, Parser_state*) /home/sbester/git/mysql-git/sql/sql_parse.cc:5582
#7 0x1ce41fd in dispatch_command(THD*, COM_DATA const*, enum_server_command) /home/sbester/git/mysql-git/sql/sql_parse.cc:1458
#8 0x1ce83c7 in do_command(THD*) /home/sbester/git/mysql-git/sql/sql_parse.cc:999
#9 0x1f7cad7 in handle_connection /home/sbester/git/mysql-git/sql/conn_handler/connection_handler_per_thread.cc:300
#10 0x2f2dbd5 in pfs_spawn_thread /home/sbester/git/mysql-git/storage/perfschema/pfs.cc:2190
#11 0x7f297d56edf4 in start_thread (/lib64/libpthread.so.0+0x7df4)
#12 0x7f297b95d60c in __clone (/lib64/libc.so.6+0xf660c)
Address 0x7f295417cae0 is located in stack of thread T30 at offset 2096 in frame
#0 0x1ea8e7f in mysql_update(THD*, List<Item>&, List<Item>&, unsigned long long, enum_duplicates, unsigned long long*, unsigned long long*) /home/sbester/git/mysql-git/sql/sql_update.cc:248
This frame has 27 object(s):
[32, 33) 'need_sort'
[96, 97) 'reverse'
[160, 164) 'dup_key_found'
[224, 228) 'result'
[288, 296) 'covering_keys_for_cond'
[352, 360) 'conds'
[416, 424) 'cond_equal'
[480, 488) 'keys_to_use'
[544, 552) 'needed_reg_dummy'
[608, 616) 'qck'
[672, 680) 'examined_rows'
[736, 744) 'found_rows'
[800, 808) 'returned_rows'
[864, 888) 'wrapper'
[928, 960) 'v'
[992, 1040) 'fsort'
[1088, 1168) 'plan'
[1216, 1296) 'plan'
[1344, 1424) 'plan'
[1472, 1552) 'plan'
[1600, 1680) 'plan'
[1728, 1848) 'update'
[1888, 2040) 'info'
[2080, 2360) 'tempfile' <== Memory access at offset 2096 is inside this variable
[2400, 2912) 'buff'
[2944, 3456) 'buff'
[3488, 4080) 'qep_tab_st'
HINT: this may be a false positive if your program uses some custom stack unwind mechanism or swapcontext
(longjmp and C++ exceptions *are* supported)
Thread T30 created by T0 here:
#0 0x7f297d9f2500 in __interceptor_pthread_create /home/sbester/build/objdir/../gcc/libsanitizer/asan/asan_interceptors.cc:243
#1 0x2f2de55 in pfs_spawn_thread_v1 /home/sbester/git/mysql-git/storage/perfschema/pfs.cc:2241
#2 0x1f7dc2b in inline_mysql_thread_create /home/sbester/git/mysql-git/include/mysql/psi/mysql_thread.h:1297
#3 0x1f7dc2b in Per_thread_connection_handler::add_connection(Channel_info*) /home/sbester/git/mysql-git/sql/conn_handler/connection_handler_per_thread.cc:403
#4 0x9519bb in Connection_handler_manager::process_new_connection(Channel_info*) /home/sbester/git/mysql-git/sql/conn_handler/connection_handler_manager.cc:268
#5 0x8c5602 in Connection_acceptor<Mysqld_socket_listener>::connection_event_loop() /home/sbester/git/mysql-git/sql/conn_handler/connection_acceptor.h:68
#6 0x8c5602 in mysqld_main(int, char**) /home/sbester/git/mysql-git/sql/mysqld.cc:5047
#7 0x7f297b888af4 in __libc_start_main /usr/src/debug/glibc-2.17-c758a686/csu/libc-start.c:274
SUMMARY: AddressSanitizer: stack-use-after-scope /home/sbester/git/mysql-git/mysys/mf_iocache.c:328 in reinit_io_cache
Shadow bytes around the buggy address:
0x0fe5aa827900: 00 00 00 00 00 00 00 00 f2 f2 f2 f2 f2 f2 00 00
0x0fe5aa827910: 00 00 00 00 00 00 00 00 f2 f2 f2 f2 f2 f2 00 00
0x0fe5aa827920: 00 00 00 00 00 00 00 00 f2 f2 f2 f2 f2 f2 00 00
0x0fe5aa827930: 00 00 00 00 00 00 00 00 00 00 00 00 00 f2 f2 f2
0x0fe5aa827940: f2 f2 00 00 00 00 00 00 00 00 00 00 00 00 00 00
=>0x0fe5aa827950: 00 00 00 00 00 f2 f2 f2 f2 f2 f8 f8[f8]f8 f8 f8
0x0fe5aa827960: f8 f8 f8 f8 f8 f8 f8 f8 f8 f8 f8 f8 f8 f8 f8 f8
0x0fe5aa827970: f8 f8 f8 f8 f8 f8 f8 f8 f8 f8 f8 f8 f8 f2 f2 f2
0x0fe5aa827980: f2 f2 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x0fe5aa827990: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x0fe5aa8279a0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
Shadow byte legend (one shadow byte represents 8 application bytes):
Addressable: 00
Partially addressable: 01 02 03 04 05 06 07
Heap left redzone: fa
Freed heap region: fd
Stack left redzone: f1
Stack mid redzone: f2
Stack right redzone: f3
Stack after return: f5
Stack use after scope: f8
Global redzone: f9
Global init order: f6
Poisoned by user: f7
Container overflow: fc
Array cookie: ac
Intra object redzone: bb
ASan internal: fe
Left alloca redzone: ca
Right alloca redzone: cb
==3829==ABORTING
[4 Nov 2017 9:43]
MySQL Verification Team
this interferes with me making a testcase on 5.7 for another tricky bug ;-/
[8 Nov 2017 12:36]
Tor Didriksen
Posted by developer: OK, with Clang on Mac, I am able to reproduce.
[13 Nov 2017 17:57]
Paul DuBois
Posted by developer: Fixed in 5.7.21. Code cleanup. No changelog entry needed.
[18 Feb 2021 9:30]
Erlend Dahl
Bug#91603 stack-use-after-scope in reinit_io_cache() detected by ASan was marked as a duplicate
Description: In the mysql_update() function, there is an IO_CACHE variable "tempfile". After having been (re)initialized, the table->sort.io_cache is set to this variable. However, the current_pos and current_end members of IO_CACHE are still pointing to values within the "tempfile" variable, which goes out of scope. This results in my_b_seek() accessing memory which is out-of-scope (and wrong in any case). tl;dr: Copying the contents of an IO_CACHE is a bad idea, and works — at best — by chance. How to repeat: Much of the test suite (for example, main.update) fails with clang's Address Sanitizer's -fsanitize-address-use-after-scope option. e.g.: ==766832==ERROR: AddressSanitizer: stack-use-after-scope on address 0x7f5fa72e46e0 at pc 0x000002821369 bp 0x7f5fa72e3930 sp 0x7f5fa72e3928 READ of size 8 at 0x7f5fa72e46e0 thread T24 #0 0x2821368 in reinit_io_cache mysys/mf_iocache.c:328:22 #1 0x1556ab7 in init_read_record(READ_RECORD*, THD*, TABLE*, QEP_TAB*, int, bool, bool) sql/records.cc:248:5 #2 0x1e49875 in mysql_update(THD*, List<Item>&, List<Item>&, unsigned long long, enum_duplicates, unsigned long long*, unsigned long long*) sql/sql_update.cc:764:17 #3 0x1e561e2 in Sql_cmd_update::try_single_table_update(THD*, bool*) sql/sql_update.cc:2879:21 #4 0x1e569f9 in Sql_cmd_update::execute(THD*) sql/sql_update.cc:3010:7 #5 0x128cee6 in mysql_execute_command(THD*, bool) sql/sql_parse.cc:3529:26 #6 0x1289728 in mysql_parse(THD*, Parser_state*) sql/sql_parse.cc:5527:20 #7 0x128673b in dispatch_command(THD*, COM_DATA const*, enum_server_command) sql/sql_parse.cc:1428:5 #8 0x1288284 in do_command(THD*) sql/sql_parse.cc:996:17 #9 0x1e1c86b in handle_connection sql/conn_handler/connection_handler_per_thread.cc:301:13 #10 0x2620b8e in pfs_spawn_thread storage/perfschema/pfs.cc:2192:3 #11 0x7f5fdda4f83f in start_thread (libpthread.so.0+0x683f) #12 0x7f5fdd1a32ec in __clone (libc.so.6+0x372ec) Address 0x7f5fa72e46e0 is located in stack of thread T24 at offset 2880 in frame #0 0x1e472bf in mysql_update(THD*, List<Item>&, List<Item>&, unsigned long long, enum_duplicates, unsigned long long*, unsigned long long*) sql/sql_update.cc:248 Suggested fix: Rework the function to not copy the contents of an IO_CACHE. For example, tempfile could be made a pointer to an IO_CACHE initially. At the very least, call "setup_io_cache(table->sort.io_cache);" after setting said variable, which will setup these particular pointers correctly.