Bug #115780 InnoDB aborts when a DB_OUT_OF_RESOURCES error occurs in Parallel_reader
Submitted: 6 Aug 2024 7:25 Modified: 7 Aug 2024 10:08
Reporter: Zhejun Cai Email Updates:
Status: Verified Impact on me:
None 
Category:MySQL Server: InnoDB storage engine Severity:S2 (Serious)
Version:8,9, 8.0.39 OS:Any
Assigned to: CPU Architecture:Any

[6 Aug 2024 7:25] Zhejun Cai
Description:
2024-08-06T07:07:17.767047Z 8 [Warning] [MY-013526] [InnoDB] Resource not available to create threads for parallel scan. Falling back to single thread mode.
2024-08-06T07:07:17.769890Z 8 [ERROR] [MY-013183] [InnoDB] Assertion failure: row0pread.h:301:active >= n_threads thread 140271021946432
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: http://dev.mysql.com/doc/refman/9.0/en/forcing-innodb-recovery.html
InnoDB: about forcing recovery.
2024-08-06T07:07:17Z UTC - mysqld got signal 6 ;
Most likely, you have hit a bug, but this error can also be caused by malfunctioning hardware.
BuildID[sha1]=ede600536a1d3802d7b869e2ef631952e864d02b
Thread pointer: 0x7f92f8001050
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 7f93646f69d8 thread_stack 0x100000

 #0 0x560d1772cb75 print_fatal_signal at /home/caizj/source/mysql-server/sql/signal_handler.cc:319
 #1 0x560d1772cf07 _Z15my_server_abortv at /home/caizj/source/mysql-server/sql/signal_handler.cc:445
 #2 0x560d18d9ff85 _Z8my_abortv at /home/caizj/source/mysql-server/mysys/my_init.cc:262
 #3 0x560d1920d222 _Z23ut_dbg_assertion_failedPKcS0_m at /home/caizj/source/mysql-server/storage/innobase/ut/ut0dbg.cc:100
 #4 0x560d18e69400 _ZN15Parallel_reader15release_threadsEm at /home/caizj/source/mysql-server/storage/innobase/include/row0pread.h:301
 #5 0x560d190eaa59 <unknown>
 #6 0x560d190e7fb1 _ZN15Parallel_reader5spawnEm at /home/caizj/source/mysql-server/storage/innobase/row/row0pread.cc:1345
 #7 0x560d190e8063 _ZN15Parallel_reader3runEm at /home/caizj/source/mysql-server/storage/innobase/row/row0pread.cc:1363
 #8 0x560d1948f722 _ZN3ddl15Parallel_cursor4scanERSt6vectorIPNS_7BuilderEN2ut9allocatorIS3_NS4_6detail18allocator_base_pfsIS3_EEEEE at /home/caizj/source/mysql-server/storage/innobase/ddl/ddl0par-scan.cc:390
 #9 0x560d194815df _ZN3ddl6Loader22scan_and_build_indexesEv at /home/caizj/source/mysql-server/storage/innobase/ddl/ddl0loader.cc:449
 #10 0x560d19481718 _ZN3ddl6Loader9build_allEv at /home/caizj/source/mysql-server/storage/innobase/ddl/ddl0loader.cc:481
 #11 0x560d19468040 _ZN3ddl7Context5buildEv at /home/caizj/source/mysql-server/storage/innobase/ddl/ddl0ctx.cc:514
 #12 0x560d18eda143 <unknown>
 #13 0x560d18eaddaa _ZN11ha_innobase19inplace_alter_tableEP5TABLEP18Alter_inplace_infoPKN2dd5TableEPS5_ at /home/caizj/source/mysql-server/storage/innobase/handler/handler0alter.cc:1570
 #14 0x560d175c675a <unknown>
 #15 0x560d175a714b mysql_inplace_alter_table at /home/caizj/source/mysql-server/sql/sql_table.cc:13813
 #16 0x560d175b36dc _Z17mysql_alter_tableP3THDPKcS2_P14HA_CREATE_INFOP9Table_refP10Alter_info at /home/caizj/source/mysql-server/sql/sql_table.cc:17687
 #17 0x560d17cb35bf _ZN19Sql_cmd_alter_table7executeEP3THD at /home/caizj/source/mysql-server/sql/sql_alter.cc:350
 #18 0x560d174b691d _Z21mysql_execute_commandP3THDb at /home/caizj/source/mysql-server/sql/sql_parse.cc:4669
 #19 0x560d174b8e09 _Z20dispatch_sql_commandP3THDP12Parser_stateb at /home/caizj/source/mysql-server/sql/sql_parse.cc:5331
 #20 0x560d174ae972 _Z16dispatch_commandP3THDPK8COM_DATA19enum_server_command at /home/caizj/source/mysql-server/sql/sql_parse.cc:2122
 #21 0x560d174ac7c2 _Z10do_commandP3THD at /home/caizj/source/mysql-server/sql/sql_parse.cc:1466
 #22 0x560d17713434 handle_connection at /home/caizj/source/mysql-server/sql/conn_handler/connection_handler_per_thread.cc:304
 #23 0x560d19ae037a pfs_spawn_thread at /home/caizj/source/mysql-server/storage/perfschema/pfs.cc:3061
 #24 0x7f93c3af4ac2 start_thread at ./nptl/pthread_create.c:442
 #25 0x7f93c3b8684f <unknown> at sysdeps/unix/sysv/linux/x86_64/clone3.S:81
 #26 0xffffffffffffffff <unknown>

Trying to get some variables.
Some pointers may be invalid and cause the dump to abort.
Query (7f92f80f0c80): alter table test.t1 add index i2(f2)
Connection ID (thread ID): 8
Status: NOT_KILLED

The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you find out what is causing the crash.
ERROR 2013 (HY000): Lost connection to MySQL server during query
No connection. Trying to reconnect...
ERROR 2002 (HY000): Can't connect to local MySQL server through socket 'mysql.sock' (111)
ERROR:
Can't connect to the server

How to repeat:
(1) Add a DBUG_EXECUTE_IF in function Parallel_cursor::scan(file innobase/ddl/ddl0par-scan.cc)

--- a/storage/innobase/ddl/ddl0par-scan.cc
+++ b/storage/innobase/ddl/ddl0par-scan.cc
@@ -371,6 +371,7 @@ dberr_t Parallel_cursor::scan(Builders &builders) noexcept {

   if (err == DB_SUCCESS) {
     err = reader.run(n_threads);
+    DBUG_EXECUTE_IF("force_ddl_preader_out_of_resource_fail", err = DB_OUT_OF_RESOURCES;);

     if (err == DB_OUT_OF_RESOURCES) {
       ut_a(!m_single_threaded_mode);

    
(2) Build MySQL with debug and restart MySQL server
Use the following CMake flag:
    -DCMAKE_BUILD_TYPE=Debug

(3) Run the following SQL statements
create database if not exists test;
create table test.t1(f1 int, f2 int, primary key(f1));
insert into test.t1 values(1,1);
set session debug="+d,force_ddl_preader_out_of_resource_fail";
alter table test.t1 add index i2(f2);

Suggested fix:
Review the code in Parallel_cursor::scan. If reader.run(n_threads) returns DB_OUT_OF_RESOURCES, there is no need to explicitly invoke reader.release_threads(n_threads) because reader.run(0) will invoke release_threads.
[7 Aug 2024 10:08] MySQL Verification Team
Hello Zhejun Cai,

Thank you for the report and test case.
Verified as described.

regards,
Umesh
[20 Aug 2024 3:42] huahua xu
The method `Parallel_reader::release_threads` should be called as early as possible, so that other user sessions can use parallel read threads.

In addition, the bug would cause the actual working thread to exceed the maximum limit, resulting in serious issues.