Bug #94994 Memory leak detect on temptable storage engine
Submitted: 12 Apr 3:06 Modified: 15 Apr 11:54
Reporter: Zhao Jianwei Email Updates:
Status: Verified Impact on me:
None 
Category:MySQL Server: Storage Engines Severity:S2 (Serious)
Version:8.0.15 OS:Linux
Assigned to: CPU Architecture:x86

[12 Apr 3:06] Zhao Jianwei
Description:
Hi, guys

When I run test on MySQL 8.0.15,  sanitizer report some memory leak,  like below:

the root cause I guess in temptable engine, the thread_local variable "shared_block" is freed when thread exit by calling "End_thread()::~End_thread()", but sanitizer will detect memory leak before main thread exit;

Direct leak of 1048576 byte(s) in 1 object(s) allocated from:
156     #0 0x7f6ecda4dc50 in malloc (/usr/lib64/libasan.so.4+0xd9c50)
157     #1 0x598f077 in temptable::Allocator<temptable::Column>::mem_fetch_from_ram(unsigned long) storage/temptable/include/temptable/allocator.h:733
158     #2 0x598f6d8 in temptable::Allocator<temptable::Column>::mem_fetch(unsigned long) storage/temptable/include/temptable/allocator.h:656
159     #3 0x598fa5b in temptable::Allocator<temptable::Column>::block_create(unsigned long) storage/temptable/include/temptable/allocator.h:901
160     #4 0x598fe56 in temptable::Allocator<temptable::Column>::allocate(unsigned long) storage/temptable/include/temptable/allocator.h:530
#18 0x5979ac5 in temptable::Handler::create(char const*, TABLE*, HA_CREATE_INFO*, dd::Table*) storage/temptable/src/handler.cc:138
#19 0x2c8f32c in create_tmp_table_with_fallback sql/sql_tmp_table.cc:2360
#20 0x2c8fcdd in instantiate_tmp_table(THD*, TABLE*, KEY*, MI_COLUMNDEF*, MI_COLUMNDEF**, unsigned long long, bool) sql/sql_tmp_table.cc:2454
#21 0x34841b4 in TABLE_LIST::create_materialized_table(THD*) sql/sql_derived.cc:797
#22 0x2b998ab in JOIN::prepare_result() sql/sql_select.cc:1420
#23 0x2a4d321 in JOIN::exec() sql/sql_executor.cc:202
#24 0x2baaa65 in handle_query(THD*, LEX*, Query_result*, unsigned long long, unsigned long long) sql/sql_select.cc:202
#25 0x2ade496 in execute_show(THD*, TABLE_LIST*) sql/sql_parse.cc:4784
#26 0x2ae7514 in mysql_execute_command(THD*, bool) sql/sql_parse.cc:2968
#27 0x2aef8dd in mysql_parse(THD*, Parser_state*, bool) sql/sql_parse.cc:5105
#28 0x2af2281 in dispatch_command(THD*, COM_DATA const*, enum_server_command) sql/sql_parse.cc:1715
#29 0x2af4bd5 in do_command(THD*) sql/sql_parse.cc:1263

How to repeat:
See  attached  t.test

./mtr  --sanitize t/t.test

Suggested fix:
Register the destructor of thread local variable in storage engine and call it when my_thread_end();

see Attached x.diff
[12 Apr 3:07] Zhao Jianwei
mtr test case

Attachment: t.test (application/octet-stream, text), 255 bytes.

[12 Apr 3:08] Zhao Jianwei
patch file ( Already OCA :) )

Attachment: x.diff (application/octet-stream, text), 5.93 KiB.

[15 Apr 1:21] zhai weixiang
just curious: does upstream run sanitizer checking or other similar tools before releasing a new version ?
[15 Apr 5:25] Umesh Shastry
Hello Zhao Jianwei,

Thank you for the report.
I tried to reproduce this issue at our end with source build but not seeing any issues as reported. Could you please share exact cmake with options used for build, OS details, gcc version? Your OCA is not yet reflecting here and once it reflects then you may have to resend the patch via "contribution" tab in order for us to accept your contribution.

- My build was on OL7

export LD_LIBRARY_PATH=/export/umesh/utils/GCC-7.3.0/lib64
export CC=/export/umesh/utils/GCC-7.3.0/bin/gcc
export CPP=/export/umesh/utils/GCC-7.3.0/bin/cpp
export CXX=/export/umesh/utils/GCC-7.3.0/bin/c++

rm -rf bld/
mkdir bld && cd bld
rm -rf CMakeCache.txt
cmake .. \
-DBUILD_CONFIG=mysql_release                  \
-DINSTALL_LAYOUT=STANDALONE                   \
-DWITH_NDBCLUSTER_STORAGE_ENGINE=0            \
-DCMAKE_INSTALL_PREFIX=$PWD                   \
-DWITH_ASAN=ON                               \
-DWITH_UBSAN=ON                               \
-DWITH_BOOST=../boost \
-DCMAKE_BUILD_TYPE=Debug

^^ tried with/without debug builds, -DWITH_ASAN or -DWITH_UBSAN or both
make -j32
make install

cd mysql-test
./mtr  --sanitize t/t.test

^^ test passed without any issues

regards,
Umesh
[15 Apr 8:04] Zhao Jianwei
Hi, Umesh Shastry

My ENV like that:

$ /opt/rh/devtoolset-7/root/usr/bin/gcc --version
gcc (GCC) 7.3.1 20180303 (Red Hat 7.3.1-5)
Copyright (C) 2017 Free Software Foundation, Inc.

COMMON_FLAGS="-O0 -g3 -gdwarf-2 -fexceptions -fno-omit-frame-pointer -fno-strict-aliasing"
CFLAGS="$COMMON_FLAGS"
CXXFLAGS="$COMMON_FLAGS"
CC=/opt/rh/devtoolset-7/root/usr/bin/gcc
CXX=/opt/rh/devtoolset-7/root/usr/bin/g++
export CC CFLAGS CXX CXXFLAGS

rm -rf CMakeCache.txt
make clean

 cmake .                                \
     -DFORCE_INSOURCE_BUILD=ON          \
     -DCMAKE_BUILD_TYPE="Debug"         \
     -DWITH_DEBUG=1                     \
     -DENABLE_GCOV=1                    \
     -DINSTALL_LAYOUT=STANDALONE        \
     -DMYSQL_MAINTAINER_MODE=0          \
     -DWITH_EMBEDDED_SERVER=0           \
     -DWITH_EXTRA_CHARSETS=all          \
     -DWITH_ZLIB=bundled                \
     -DWITH_MYISAM_STORAGE_ENGINE=1     \
     -DWITH_INNOBASE_STORAGE_ENGINE=1   \
     -DWITH_CSV_STORAGE_ENGINE=1        \
     -DWITH_ARCHIVE_STORAGE_ENGINE=1    \
     -DWITH_BLACKHOLE_STORAGE_ENGINE=1  \
     -DWITH_FEDERATED_STORAGE_ENGINE=1  \
     -DWITH_PERFSCHEMA_STORAGE_ENGINE=1 \
     -DWITH_EXAMPLE_STORAGE_ENGINE=0    \
     -DWITH_TEMPTABLE_STORAGE_ENGINE=1   \
     -DENABLED_PROFILING=1              \
     -DENABLED_LOCAL_INFILE=1           \
     -DWITH_ASAN=1                      \
     -DWITH_BOOST="../boost_1_68_0.tar.gz" \

make -j16

./mtr --sanitize t/t.test

ATTENTION:

Please "./mtr --sanitize t/t.test  --record " to generate result file,  and retry "./mtr --sanitize t/t.test" again, then I get the sanitizer report as :

==============================================================================
                  TEST NAME                       RESULT  TIME (ms) COMMENT
------------------------------------------------------------------------------
[ 50%] main.t                                    [ pass ]     21
worker[1] Sanitizer report from /flash1/u01/jianwei.zhao/mysql-server/mysql-test/var/log/mysqld.1.err after tests:
 main.t
------------------------------------------------------------------------------
==8195==ERROR: LeakSanitizer: detected memory leaks

Direct leak of 1048576 byte(s) in 1 object(s) allocated from:
    #0 0x7faaa4893c50 in malloc (/usr/lib64/libasan.so.4+0xd9c50)
    #1 0x67f21d5 in temptable::Allocator<temptable::Column>::mem_fetch_from_ram(unsigned long) /u01/jianwei.zhao/mysql-server/storage/temptable/include/temptable/allocator.h:733
    #2 0x67f283e in temptable::Allocator<temptable::Column>::mem_fetch(unsigned long) /u01/jianwei.zhao/mysql-server/storage/temptable/include/temptable/allocator.h:656
    #3 0x67f2dc4 in temptable::Allocator<temptable::Column>::block_create(unsigned long) /u01/jianwei.zhao/mysql-server/storage/temptable/include/temptable/allocator.h:901
    #4 0x67f32b9 in temptable::Allocator<temptable::Column>::allocate(unsigned long) /u01/jianwei.zhao/mysql-server/storage/temptable/include/temptable/allocator.h:530
    #5 0x67f33d4 in std::allocator_traits<temptable::Allocator<temptable::Column> >::allocate(temptable::Allocator<temptable::Column>&, unsigned long) /opt/rh/devtoolset-7/root/usr/include/c++/7/bits/alloc_traits.h:301

....

SUMMARY: AddressSanitizer: 2097228 byte(s) leaked in 3 allocation(s).
safe_process[8194]: Child process: 8195, exit: 42

[100%] sanitize_report                           [ fail ]
        Test ended at 2019-04-15 15:58:56

Sanitizer reported failures at shutdown, see above
[15 Apr 10:50] Umesh Shastry
Thank you for the requested details, will try at my end and comeback to you if anything further required to reproduce. Thank you!
[15 Apr 11:34] Umesh Shastry
I'm able to reproduce this on Ubuntu 18.10 but not on OL7 even with the gcc 7.3.x and same flags as provided.

regards,
Umesh
[15 Apr 11:35] Umesh Shastry
8.0.15 - test results

Attachment: 94994_8.0.15.results (application/octet-stream, text), 460.44 KiB.

[15 Apr 11:54] Zhao Jianwei
Hi, Umesh Shastry

Thanks for your patience and detailed  verification.

I think all the plugin/storage engine defined thread local variables are better to deallocate before my_thread_end() than automatic free when thread exit within glibc;