| Bug #96217 | Confliction of resource group and thread pool plugin, invalid memory access | ||
|---|---|---|---|
| Submitted: | 16 Jul 2019 9:25 | Modified: | 30 Jul 2019 10:49 |
| Reporter: | Fangxin Flou (OCA) | Email Updates: | |
| Status: | Can't repeat | Impact on me: | |
| Category: | MySQL Server: Connection Handling | Severity: | S3 (Non-critical) |
| Version: | 8.0 | OS: | Any |
| Assigned to: | CPU Architecture: | Any | |
[18 Jul 2019 9:18]
Umesh Shastry
Hello Fangxin Flou, Thank you for the report and feedback. I tried to reproduce this issue at my end but not seeing any memory access reported issues after shutdown. Could you please provide exact cmake command used for the build, MySQL server version and OS details to reproduce this issue at our end? Thank you. I'm joining activity log shortly for your reference. regards, Umesh
[18 Jul 2019 9:18]
Umesh Shastry
Test results - 8.0.16
Attachment: 96217_8.0.16.results (application/octet-stream, text), 4.44 KiB.
[29 Jul 2019 2:00]
Fangxin Flou
build with -DWITH_ASAN=1 and run with performance_schema = on
[29 Jul 2019 2:01]
Fangxin Flou
the crash stack
Attachment: threadpool_resource_group_issue.png (image/png, text), 219.43 KiB.
[29 Jul 2019 5:04]
Umesh Shastry
Thank you for the feedback. Please note that I have already built using -DWITH_ASAN=ON and performance_schema is on by default. Is there anything else required to trigger the issue? Really appreciate if you can share exact MySQL version that you are using, OS, gcc version etc? Thank you. [18 Jul 9:18] Umesh Shastry Test results - 8.0.16 Attachment: 96217_8.0.16.results (application/octet-stream, text), 4.44 KiB. regards, Umesh
[29 Jul 2019 7:19]
Umesh Shastry
Worklog attempted on Debian10 - no issues observed here
Attachment: 96217_8.0.17.Debian10 (application/octet-stream, text), 2.41 MiB.
[30 Jul 2019 3:42]
Fangxin Flou
CC=/opt/rh/devtoolset-7/root/usr/bin/gcc
CXX=/opt/rh/devtoolset-7/root/usr/bin/g++
export CC CFLAGS CXX CXXFLAGS
rm -rf CMakeCache.txt
make clean
cmake . \
-DFORCE_INSOURCE_BUILD=ON \
-DCMAKE_BUILD_TYPE="$build_type" \
-DSYSCONFDIR="$dest_dir" \
-DCMAKE_INSTALL_PREFIX="$dest_dir" \
-DMYSQL_DATADIR="$dest_dir/data" \
-DWITH_DEBUG=$debug \
-DENABLE_GCOV=$gcov \
-DINSTALL_LAYOUT=STANDALONE \
-DMYSQL_MAINTAINER_MODE=0 \
-DWITH_EMBEDDED_SERVER=0 \
-DWITH_EXTRA_CHARSETS=all \
-DWITH_SSL=openssl \
-DWITH_ZLIB=bundled \
-DWITH_MYISAM_STORAGE_ENGINE=1 \
-DWITH_INNOBASE_STORAGE_ENGINE=1 \
-DWITH_CSV_STORAGE_ENGINE=1 \
-DWITH_ARCHIVE_STORAGE_ENGINE=1 \
-DWITH_BLACKHOLE_STORAGE_ENGINE=1 \
-DWITH_FEDERATED_STORAGE_ENGINE=1 \
-DWITH_PERFSCHEMA_STORAGE_ENGINE=1 \
-DWITH_EXAMPLE_STORAGE_ENGINE=0 \
-DWITH_TEMPTABLE_STORAGE_ENGINE=1 \
-DENABLED_PROFILING=1 \
-DENABLED_LOCAL_INFILE=1 \
-DWITH_ASAN=$asan \
-DWITH_TSAN=$tsan \
-DWITH_BOOST="extra/boost/boost_1_69_0.tar.gz" \
-DMYSQL_SERVER_SUFFIX="$server_suffix"
build_type="RelWithDebInfo"
debug=0
gcov=0
asan=1
tsan=0
[30 Jul 2019 10:49]
Umesh Shastry
Thank you for the cmake details but sorry this is not reproducible at our end even with the provided cmake options. Since requested OS, MySQL server version details are not available hence I'm not sure I can do much here unless provided. Joining worklog carried out on CentOS 7 shorty. regards, Umesh
[30 Jul 2019 10:50]
Umesh Shastry
Test results - 8.0.18
Attachment: 96217_8.0.18.results (application/octet-stream, text), 36.06 KiB.
Description: We have a plugin named "thread_pool", when loaded resource group will be disabled by following code in mysqld.cc LEX_CSTRING plugin_name = {C_STRING_WITH_LEN("thread_pool")}; if (Connection_handler_manager::thread_handling != Connection_handler_manager::SCHEDULER_ONE_THREAD_PER_CONNECTION || plugin_is_ready(plugin_name, MYSQL_DAEMON_PLUGIN)) { auto res_grp_mgr = resourcegroups::Resource_group_mgr::instance(); res_grp_mgr->disable_resource_group(); res_grp_mgr->set_unsupport_reason("Thread pool plugin enabled"); } when disabled, it's possible some threads (for example the pfs thread) have created before resource group disable. the deinit of disable_resource_group will release some memory structures for thread termination which cause the invalid memory access. How to repeat: Load a daemon plugin named "thread_pool", and compiled with libasan, will get the following errors during shutdown. heap_use_after_free ... ... pfs_spawn_thread ... ... Suggested fix: remove the following code in mysqld.cc LEX_CSTRING plugin_name = {C_STRING_WITH_LEN("thread_pool")}; if (Connection_handler_manager::thread_handling != Connection_handler_manager::SCHEDULER_ONE_THREAD_PER_CONNECTION || plugin_is_ready(plugin_name, MYSQL_DAEMON_PLUGIN)) { auto res_grp_mgr = resourcegroups::Resource_group_mgr::instance(); res_grp_mgr->disable_resource_group(); res_grp_mgr->set_unsupport_reason("Thread pool plugin enabled"); } and skip the resource logic for thread pool connection in sql_parse.cc if (thd->scheduler.data == NULL && thd->resource_group_ctx()->m_warn != 0) { auto res_grp_name = thd->resource_group_ctx()->m_switch_resource_group_str; switch (thd->resource_group_ctx()->m_warn) { case WARN_RESOURCE_GROUP_UNSUPPORTED: { auto res_grp_mgr = resourcegroups::Resource_group_mgr::instance(); push_warning_printf(thd, Sql_condition::SL_WARNING, ER_FEATURE_UNSUPPORTED, ER_THD(thd, ER_FEATURE_UNSUPPORTED), "Resource groups", res_grp_mgr->unsupport_reason()); break; } ......