| Bug #109540 | heap-buffer-overflow in StrXfrmTest.ChineseUTF8MB4 gunit test | ||
|---|---|---|---|
| Submitted: | 5 Jan 2023 16:23 | Modified: | 18 Apr 2023 18:55 |
| Reporter: | Yura Sorokin (OCA) | Email Updates: | |
| Status: | Closed | Impact on me: | |
| Category: | MySQL Server: Charsets | Severity: | S7 (Test Cases) |
| Version: | 8.0.31 | OS: | Any |
| Assigned to: | CPU Architecture: | Any | |
[6 Jan 2023 8:07]
MySQL Verification Team
Hi, We do not support Percona Server. Please confirm if MySQL Server from dev.mysql.com is affected too.
[6 Jan 2023 8:43]
Yura Sorokin
Yes,I confirm, the problem exists in the upstream (MySQL Server 8.0.31) as well. The stack traces look exactly the same.
[9 Jan 2023 12:42]
MySQL Verification Team
Hi Mr. Sorokin, Thank you very much for your bug report. Would you be so kind to send us a full stacktrace from our MySQL 8.0.31 server. Also, we need a fully repeatable test case in order to proceed with processing of your report. We are waiting on your feedback.
[11 Jan 2023 8:46]
Yura Sorokin
Just run merge_small_tests-t compiled with gcc-12 ASan with the following set of tests ./merge_small_tests-t --gtest_filter=CharsetBug32788301Unittest.LoadUninitLoad:Microbenchmarks.BM_Chinese_AS_CS:StrXfrmTest.ChineseUTF8MB4 Note: Google Test filter = CharsetBug32788301Unittest.LoadUninitLoad:Microbenchmarks.BM_Chinese_AS_CS:StrXfrmTest.ChineseUTF8MB4 [==========] Running 3 tests from 3 test suites. [----------] Global test environment set-up. [----------] 1 test from Microbenchmarks [ RUN ] Microbenchmarks.BM_Chinese_AS_CS WARNING: Running microbenchmark in debug mode. Timings will be misleading. BM_Chinese_AS_CS 10 iterations 387896 ns/iter 1.18 MB/sec [ OK ] Microbenchmarks.BM_Chinese_AS_CS (435 ms) [----------] 1 test from Microbenchmarks (435 ms total) [----------] 1 test from CharsetBug32788301Unittest [ RUN ] CharsetBug32788301Unittest.LoadUninitLoad [ OK ] CharsetBug32788301Unittest.LoadUninitLoad (135 ms) [----------] 1 test from CharsetBug32788301Unittest (135 ms total) [----------] 1 test from StrXfrmTest [ RUN ] StrXfrmTest.ChineseUTF8MB4
[11 Jan 2023 8:47]
Yura Sorokin
=================================================================
==601055==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x62000009ce90 at pc 0x5595b26dba8d bp 0x7ffc031d6030 sp 0x7ffc031d6020
READ of size 2 at 0x62000009ce90 thread T0
#0 0x5595b26dba8c in modify_all_zh_pages /home/yura/ws/mysql-server/strings/ctype-uca.cc:4180
#1 0x5595b26dc6c7 in init_weight_level /home/yura/ws/mysql-server/strings/ctype-uca.cc:4289
#2 0x5595b26dcfac in create_tailoring /home/yura/ws/mysql-server/strings/ctype-uca.cc:4798
#3 0x5595b26dd56d in my_coll_init_uca /home/yura/ws/mysql-server/strings/ctype-uca.cc:4845
#4 0x5595b25ea9e6 in get_internal_charset /home/yura/ws/mysql-server/mysys/charset.cc:561
#5 0x5595b25ebc9b in my_collation_get_by_name(MY_CHARSET_LOADER*, char const*, int) /home/yura/ws/mysql-server/mysys/charset.cc:612
#6 0x5595b250a987 in init_collation /home/yura/ws/mysql-server/unittest/gunit/strings_strnxfrm-t.cc:93
#7 0x5595b2510d63 in strnxfrm_unittest::StrXfrmTest_ChineseUTF8MB4_Test::TestBody() /home/yura/ws/mysql-server/unittest/gunit/strings_strnxfrm-t.cc:1785
#8 0x5595b279e0d2 in void testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /home/yura/ws/mysql-server/extra/googletest/googletest-release-1.12.0/googletest/src/gtest.cc:2599
#9 0x5595b27c7b8f in void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /home/yura/ws/mysql-server/extra/googletest/googletest-release-1.12.0/googletest/src/gtest.cc:2635
#10 0x5595b278e8ee in testing::Test::Run() /home/yura/ws/mysql-server/extra/googletest/googletest-release-1.12.0/googletest/src/gtest.cc:2674
#11 0x5595b278ecea in testing::TestInfo::Run() /home/yura/ws/mysql-server/extra/googletest/googletest-release-1.12.0/googletest/src/gtest.cc:2853
#12 0x5595b278efc4 in testing::TestSuite::Run() /home/yura/ws/mysql-server/extra/googletest/googletest-release-1.12.0/googletest/src/gtest.cc:3012
#13 0x5595b2796eb4 in testing::internal::UnitTestImpl::RunAllTests() /home/yura/ws/mysql-server/extra/googletest/googletest-release-1.12.0/googletest/src/gtest.cc:5870
#14 0x5595b279f2be in bool testing::internal::HandleSehExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /home/yura/ws/mysql-server/extra/googletest/googletest-release-1.12.0/googletest/src/gtest.cc:2599
#15 0x5595b27c895a in bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /home/yura/ws/mysql-server/extra/googletest/googletest-release-1.12.0/googletest/src/gtest.cc:2635
#16 0x5595b278f2d1 in testing::UnitTest::Run() /home/yura/ws/mysql-server/extra/googletest/googletest-release-1.12.0/googletest/src/gtest.cc:5444
#17 0x5595b25e4266 in RUN_ALL_TESTS() /home/yura/ws/mysql-server/extra/googletest/googletest-release-1.12.0/googletest/include/gtest/gtest.h:2293
#18 0x5595b25e3f72 in main /home/yura/ws/mysql-server/unittest/gunit/gunit_test_main.cc:150
#19 0x7f22058a5d8f in __libc_start_call_main ../sysdeps/nptl/libc_start_call_main.h:58
#20 0x7f22058a5e3f in __libc_start_main_impl ../csu/libc-start.c:392
#21 0x5595b2080b44 in _start (/home/yura/ws/mysql-build-8.0-asan_gcc12/runtime_output_directory/merge_small_tests-t+0x686b44)
[11 Jan 2023 8:47]
Yura Sorokin
0x62000009ce90 is located 0 bytes to the right of 3600-byte region [0x62000009c080,0x62000009ce90)
allocated by thread T0 here:
#0 0x7f220644b90f in __interceptor_malloc ../../../../src/libsanitizer/asan/asan_malloc_linux.cpp:69
#1 0x5595b261d147 in my_once_alloc(unsigned long, int) /home/yura/ws/mysql-server/mysys/my_once.cc:80
#2 0x5595b25e8796 in MY_CHARSET_LOADER::once_alloc(unsigned long) /home/yura/ws/mysql-server/mysys/charset.cc:336
#3 0x5595b2671a13 in my_uca_copy_page /home/yura/ws/mysql-server/strings/ctype-uca.cc:3728
#4 0x5595b26dc688 in init_weight_level /home/yura/ws/mysql-server/strings/ctype-uca.cc:4286
#5 0x5595b26dcfac in create_tailoring /home/yura/ws/mysql-server/strings/ctype-uca.cc:4798
#6 0x5595b26dd56d in my_coll_init_uca /home/yura/ws/mysql-server/strings/ctype-uca.cc:4845
#7 0x5595b25ea9e6 in get_internal_charset /home/yura/ws/mysql-server/mysys/charset.cc:561
#8 0x5595b25ebc9b in my_collation_get_by_name(MY_CHARSET_LOADER*, char const*, int) /home/yura/ws/mysql-server/mysys/charset.cc:612
#9 0x5595b250a987 in init_collation /home/yura/ws/mysql-server/unittest/gunit/strings_strnxfrm-t.cc:93
#10 0x5595b2510d63 in strnxfrm_unittest::StrXfrmTest_ChineseUTF8MB4_Test::TestBody() /home/yura/ws/mysql-server/unittest/gunit/strings_strnxfrm-t.cc:1785
#11 0x5595b279e0d2 in void testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /home/yura/ws/mysql-server/extra/googletest/googletest-release-1.12.0/googletest/src/gtest.cc:2599
#12 0x5595b27c7b8f in void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /home/yura/ws/mysql-server/extra/googletest/googletest-release-1.12.0/googletest/src/gtest.cc:2635
#13 0x5595b278e8ee in testing::Test::Run() /home/yura/ws/mysql-server/extra/googletest/googletest-release-1.12.0/googletest/src/gtest.cc:2674
#14 0x5595b278ecea in testing::TestInfo::Run() /home/yura/ws/mysql-server/extra/googletest/googletest-release-1.12.0/googletest/src/gtest.cc:2853
#15 0x5595b278efc4 in testing::TestSuite::Run() /home/yura/ws/mysql-server/extra/googletest/googletest-release-1.12.0/googletest/src/gtest.cc:3012
#16 0x5595b2796eb4 in testing::internal::UnitTestImpl::RunAllTests() /home/yura/ws/mysql-server/extra/googletest/googletest-release-1.12.0/googletest/src/gtest.cc:5870
#17 0x5595b279f2be in bool testing::internal::HandleSehExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /home/yura/ws/mysql-server/extra/googletest/googletest-release-1.12.0/googletest/src/gtest.cc:2599
#18 0x5595b27c895a in bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /home/yura/ws/mysql-server/extra/googletest/googletest-release-1.12.0/googletest/src/gtest.cc:2635
#19 0x5595b278f2d1 in testing::UnitTest::Run() /home/yura/ws/mysql-server/extra/googletest/googletest-release-1.12.0/googletest/src/gtest.cc:5444
#20 0x5595b25e4266 in RUN_ALL_TESTS() /home/yura/ws/mysql-server/extra/googletest/googletest-release-1.12.0/googletest/include/gtest/gtest.h:2293
#21 0x5595b25e3f72 in main /home/yura/ws/mysql-server/unittest/gunit/gunit_test_main.cc:150
#22 0x7f22058a5d8f in __libc_start_call_main ../sysdeps/nptl/libc_start_call_main.h:58
SUMMARY: AddressSanitizer: heap-buffer-overflow /home/yura/ws/mysql-server/strings/ctype-uca.cc:4180 in modify_all_zh_pages
Shadow bytes around the buggy address:
0x0c408000b980: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x0c408000b990: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x0c408000b9a0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x0c408000b9b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x0c408000b9c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
=>0x0c408000b9d0: 00 00[fa]fa fa fa fa fa fa fa fa fa fa fa fa fa
0x0c408000b9e0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
0x0c408000b9f0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
0x0c408000ba00: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
0x0c408000ba10: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x0c408000ba20: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
Shadow byte legend (one shadow byte represents 8 application bytes):
Addressable: 00
Partially addressable: 01 02 03 04 05 06 07
Heap left redzone: fa
Freed heap region: fd
Stack left redzone: f1
Stack mid redzone: f2
Stack right redzone: f3
Stack after return: f5
Stack use after scope: f8
Global redzone: f9
Global init order: f6
Poisoned by user: f7
Container overflow: fc
Array cookie: ac
Intra object redzone: bb
ASan internal: fe
Left alloca redzone: ca
Right alloca redzone: cb
==601055==ABORTING
[11 Jan 2023 13:18]
MySQL Verification Team
Hi Mr. Sorokin, You are correct, there are memory leaks. However, these are small errors in the client unit tests, which makes this bug a very low priority one. Verified.
[18 Apr 2023 18:55]
Jon Stephens
Documented fix as follows in the MySQL 8.1.0 changelog:
The loading and unloading of UCA character sets has been
rewritten to improve memory handling during repeated cycles of
initialization and de-initialization.
Closed.
[19 Apr 2023 12:33]
MySQL Verification Team
Thank you, Jon.
[19 Jul 2023 21:58]
Jon Stephens
This also fixes BUG#110836.
[29 Feb 11:37]
MySQL Verification Team
Bug #113709 marked as duplicate of this one
[29 Feb 11:42]
MySQL Verification Team
That means that this bug is fixed in 8.1 and higher, but not in 8.0.
Description: Running merge_small_tests-t under ASan generates the following error [----------] 1 test from StrXfrmTest [ RUN ] StrXfrmTest.ChineseUTF8MB4 ================================================================= ==310939==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x62000009ce90 at pc 0x55f05d116a09 bp 0x7ffe8d1e6f60 sp 0x7ffe8d1e6f50 READ of size 2 at 0x62000009ce90 thread T0 #0 0x55f05d116a08 in modify_all_zh_pages /home/yura/ws/percona-server/strings/ctype-uca.cc:4194 #1 0x55f05d117643 in init_weight_level /home/yura/ws/percona-server/strings/ctype-uca.cc:4303 #2 0x55f05d117f28 in create_tailoring /home/yura/ws/percona-server/strings/ctype-uca.cc:4812 #3 0x55f05d1184e9 in my_coll_init_uca /home/yura/ws/percona-server/strings/ctype-uca.cc:4859 #4 0x55f05d0222d4 in get_internal_charset /home/yura/ws/percona-server/mysys/charset.cc:577 #5 0x55f05d023589 in my_collation_get_by_name(MY_CHARSET_LOADER*, char const*, int) /home/yura/ws/percona-server/mysys/charset.cc:628 #6 0x55f05cf41bed in init_collation /home/yura/ws/percona-server/unittest/gunit/strings_strnxfrm-t.cc:93 #7 0x55f05cf47fc9 in strnxfrm_unittest::StrXfrmTest_ChineseUTF8MB4_Test::TestBody() /home/yura/ws/percona-server/unittest/gunit/strings_strnxfrm-t.cc:1785 ... 0x62000009ce90 is located 0 bytes to the right of 3600-byte region [0x62000009c080,0x62000009ce90) allocated by thread T0 here: #0 0x7f7d95be090f in __interceptor_malloc ../../../../src/libsanitizer/asan/asan_malloc_linux.cpp:69 #1 0x55f05d055ab3 in my_once_alloc(unsigned long, int) /home/yura/ws/percona-server/mysys/my_once.cc:80 #2 0x55f05d020052 in MY_CHARSET_LOADER::once_alloc(unsigned long) /home/yura/ws/percona-server/mysys/charset.cc:336 #3 0x55f05d0abb47 in my_uca_copy_page /home/yura/ws/percona-server/strings/ctype-uca.cc:3742 #4 0x55f05d117604 in init_weight_level /home/yura/ws/percona-server/strings/ctype-uca.cc:4300 #5 0x55f05d117f28 in create_tailoring /home/yura/ws/percona-server/strings/ctype-uca.cc:4812 #6 0x55f05d1184e9 in my_coll_init_uca /home/yura/ws/percona-server/strings/ctype-uca.cc:4859 #7 0x55f05d0222d4 in get_internal_charset /home/yura/ws/percona-server/mysys/charset.cc:577 #8 0x55f05d023589 in my_collation_get_by_name(MY_CHARSET_LOADER*, char const*, int) /home/yura/ws/percona-server/mysys/charset.cc:628 #9 0x55f05cf41bed in init_collation /home/yura/ws/percona-server/unittest/gunit/strings_strnxfrm-t.cc:93 #10 0x55f05cf47fc9 in strnxfrm_unittest::StrXfrmTest_ChineseUTF8MB4_Test::TestBody() /home/yura/ws/percona-server/unittest/gunit/strings_strnxfrm-t.cc:1785 ... SUMMARY: AddressSanitizer: heap-buffer-overflow /home/yura/ws/percona-server/strings/ctype-uca.cc:4194 in modify_all_zh_pages Shadow bytes around the buggy address: 0x0c408000b980: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 0x0c408000b990: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 0x0c408000b9a0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 0x0c408000b9b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 0x0c408000b9c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 =>0x0c408000b9d0: 00 00[fa]fa fa fa fa fa fa fa fa fa fa fa fa fa 0x0c408000b9e0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa 0x0c408000b9f0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa 0x0c408000ba00: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa 0x0c408000ba10: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 0x0c408000ba20: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 Shadow byte legend (one shadow byte represents 8 application bytes): Addressable: 00 Partially addressable: 01 02 03 04 05 06 07 Heap left redzone: fa Freed heap region: fd Stack left redzone: f1 Stack mid redzone: f2 Stack right redzone: f3 Stack after return: f5 Stack use after scope: f8 Global redzone: f9 Global init order: f6 Poisoned by user: f7 Container overflow: fc Array cookie: ac Intra object redzone: bb ASan internal: fe Left alloca redzone: ca Right alloca redzone: cb ==310939==ABORTING How to repeat: Build MySQL Server (with unit tests enabled) with gcc-12 with ASan enabled on Ubunty 22.04 Jammy cmake ... -DCMAKE_BUILD_TYPE=Debug -DCMAKE_C_COMPILER=gcc-12 -DCMAKE_CXX_COMPILER=g++-12 -DWITH_PACKAGE_FLAGS=OFF -DWITH_ASAN=ON -DWITH_ASAN_SCOPE=ON ./merge_small_tests-t Suggested fix: Narrowing the problem down showed that heap-buffer-overflow does not happen when StrXfrmTest.ChineseUTF8MB4 is run separately. The issue exists only when this unit test is run after Microbenchmarks.BM_Chinese_AS_CS and CharsetBug32788301Unittest.LoadUninitLoad in this particular order. In other words ./merge_small_tests-t --gtest_shuffle --gtest_random_seed=2 --gtest_filter=CharsetBug32788301Unittest.LoadUninitLoad:Microbenchmarks.BM_Chinese_AS_CS:StrXfrmTest.ChineseUTF8MB4 Additional investigation revealed that the problem is with init_collation("utf8mb4_zh_0900_as_cs"); run after charset_uninit(); Basically, the following new gunit test added to '/unittest/gunit/strings_strnxfrm-t.cc' generates the same ASan error TEST(Blah, Foo) { CHARSET_INFO *csa = init_collation("utf8mb4_zh_0900_as_cs"); charset_uninit(); CHARSET_INFO *csb = init_collation("utf8mb4_zh_0900_as_cs"); } It looks like the issue is with calling create_tailoring() for the same charset after calling charset_uninit(). The issue may be similar / related to Bug #32788301"UNIT TEST MERGE_SMALL_TESTS INSTABILITY ON PB2" and Bug #28956360 "CRASH WHEN CALLING MYSQL_REAL_CONNECT() IN LOOP" https://github.com/mysql/mysql-server/blob/8.0/unittest/gunit/charset_bug32788301-t.cc https://github.com/mysql/mysql-server/blob/8.0/unittest/gunit/charset_bug28956360-t.cc