Bug #107216 InnoDB: Assertion failure in thread 139807253513984 in file rem0rec.cc line 578
Submitted: 5 May 2022 8:20 Modified: 5 May 2022 12:10
Reporter: snow tu Email Updates:
Status: Unsupported Impact on me:
None 
Category:MySQL Server Severity:S3 (Non-critical)
Version:5.7.19 OS:Any
Assigned to: CPU Architecture:Any
Tags: assertion failure, in file rem0rec.cc, innodb

[5 May 2022 8:20] snow tu
Description:
One of our MySQL instance(5.7.19) crashed.

(1)MySQL  error log as bellow:
2022-05-02 07:00:02 0x7f2769aec700  InnoDB: Assertion failure in thread 139807253513984 in file rem0rec.cc line 578
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: http://dev.mysql.com/doc/refman/5.7/en/forcing-innodb-recovery.html
InnoDB: about forcing recovery.
23:00:02 UTC - mysqld got signal 6 ;
........

Thread pointer: 0x7f27685ac000
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 7f2769aebbe8 thread_stack 0x80000
/usr/local/mysql/bin/mysqld(my_print_stacktrace+0x2c)[0xed36bc]
/usr/local/mysql/bin/mysqld(handle_fatal_signal+0x461)[0x7a26e1]
/lib64/libpthread.so.0[0x369ea0f7e0]
/lib64/libc.so.6(gsignal+0x35)[0x369e232625]
/lib64/libc.so.6(abort+0x175)[0x369e233e05]
/usr/local/mysql/bin/mysqld[0x770b56]
/usr/local/mysql/bin/mysqld(_Z20rec_get_offsets_funcPKhPK12dict_index_tPmmPP16mem_block_info_t+0x56)[0x1004376]
/usr/local/mysql/bin/mysqld(_Z15row_search_mvccPh15page_cur_mode_tP14row_prebuilt_tmm+0x2393)[0x105e183]
/usr/local/mysql/bin/mysqld(_ZN11ha_innobase13general_fetchEPhjj+0x1b8)[0xf4c158]
/usr/local/mysql/bin/mysqld(_ZN7handler13ha_index_nextEPh+0x1a4)[0x8059a4]
/usr/local/mysql/bin/mysqld(_ZN7handler15read_range_nextEv+0x20)[0x809dd0]
/usr/local/mysql/bin/mysqld(_ZN7handler21multi_range_read_nextEPPc+0xd2)[0x7ff3b2]
/usr/local/mysql/bin/mysqld(_ZN18QUICK_RANGE_SELECT8get_nextEv+0x4b)[0xdd9c0b]
/usr/local/mysql/bin/mysqld[0xc1464a]
/usr/local/mysql/bin/mysqld(_ZN14Sql_cmd_delete12mysql_deleteEP3THDy+0x1162)[0xe130e2]
/usr/local/mysql/bin/mysqld(_ZN14Sql_cmd_delete7executeEP3THD+0xbf)[0xe1378f]
/usr/local/mysql/bin/mysqld(_Z21mysql_execute_commandP3THDb+0x18f0)[0xcad230]
/usr/local/mysql/bin/mysqld(_Z11mysql_parseP3THDP12Parser_state+0x5d5)[0xcb3385]
/usr/local/mysql/bin/mysqld(_Z16dispatch_commandP3THDPK8COM_DATA19enum_server_command+0xaba)[0xcb3eca]
/usr/local/mysql/bin/mysqld(_Z10do_commandP3THD+0x1b7)[0xcb5917]
/usr/local/mysql/bin/mysqld(_Z26threadpool_process_requestP3THD+0xc7)[0xd5a267]
/usr/local/mysql/bin/mysqld[0xd6979e]
/usr/local/mysql/bin/mysqld(pfs_spawn_thread+0x1b4)[0x123cf24]
/lib64/libpthread.so.0[0x369ea07aa1]
/lib64/libc.so.6(clone+0x6d)[0x369e2e893d]

Trying to get some variables.
Some pointers may be invalid and cause the dump to abort.
Query (7f2762265030): is an invalid pointer
Connection ID (thread ID): 6327781
Status: NOT_KILLED

(2)MySQL core bt as bellow:
(gdb) bt 
#0  0x000000369ea0c97c in pthread_kill () from /lib64/libpthread.so.0
#1  0x00000000007a2655 in handle_fatal_signal (sig=6) at /mnt/workspace/percona-server-5.7-binaries-release-rocks/label_exp/centos6-64/percona-server-5.7.19-17/sql/signal_handler.cc:223
#2  <signal handler called>
#3  0x000000369e232625 in raise () from /lib64/libc.so.6
#4  0x000000369e233e05 in abort () from /lib64/libc.so.6
#5  0x0000000000770b56 in ut_dbg_assertion_failed (expr=0x0, file=Unhandled dwarf expression opcode 0xf3) 
	at /mnt/workspace/percona-server-5.7-binaries-release-rocks/label_exp/centos6-64/percona-server-5.7.19-17/storage/innobase/ut/ut0dbg.cc:67
#6  0x0000000001004376 in rec_get_offsets_func (rec=0x7f28a0ea7ce2 "dipwd.suning.comNJYH\342n\372\376\200", index=0x7f278f9bad88, =Unhandled dwarf expression opcode 0xf3) 
	at /mnt/workspace/percona-server-5.7-binaries-release-rocks/label_exp/centos6-64/percona-server-5.7.19-17/storage/innobase/rem/rem0rec.cc:578
#7  0x000000000105e183 in row_search_mvcc (buf=Unhandled dwarf expression opcode 0xf3)
    at /mnt/workspace/percona-server-5.7-binaries-release-rocks/label_exp/centos6-64/percona-server-5.7.19-17/storage/innobase/row/row0sel.cc:5255
#8  0x0000000000f4c158 in ha_innobase::general_fetch (this=0x7f278f2c0a30, buf=0x7f2762a66730 "", direction=1, match_mode=0)
    at /mnt/workspace/percona-server-5.7-binaries-release-rocks/label_exp/centos6-64/percona-server-5.7.19-17/storage/innobase/handler/ha_innodb.cc:9788
#9  0x00000000008059a4 in handler::ha_index_next (this=0x7f278f2c0a30, buf=0x7f2762a66730 "")
    at /mnt/workspace/percona-server-5.7-binaries-release-rocks/label_exp/centos6-64/percona-server-5.7.19-17/sql/handler.cc:3353
#10 0x0000000000809dd0 in handler::read_range_next (this=0x7f278f2c0a30)
    at /mnt/workspace/percona-server-5.7-binaries-release-rocks/label_exp/centos6-64/percona-server-5.7.19-17/sql/handler.cc:7764
#11 0x00000000007ff3b2 in handler::multi_range_read_next (this=0x7f278f2c0a30, range_info=0x7f2769ae9520)
    at /mnt/workspace/percona-server-5.7-binaries-release-rocks/label_exp/centos6-64/percona-server-5.7.19-17/sql/handler.cc:6766
#12 0x0000000000dd9c0b in QUICK_RANGE_SELECT::get_next (this=0x7f275f59f400)
    at /mnt/workspace/percona-server-5.7-binaries-release-rocks/label_exp/centos6-64/percona-server-5.7.19-17/sql/opt_range.cc:11237
#13 0x0000000000c1464a in rr_quick (info=0x7f2769ae9690) at /mnt/workspace/percona-server-5.7-binaries-release-rocks/label_exp/centos6-64/percona-server-5.7.19-17/sql/records.cc:398
#14 0x0000000000e130e2 in Sql_cmd_delete::mysql_delete (this=Unhandled dwarf expression opcode 0xf3)
    at /mnt/workspace/percona-server-5.7-binaries-release-rocks/label_exp/centos6-64/percona-server-5.7.19-17/sql/sql_delete.cc:452
#15 0x0000000000e1378f in Sql_cmd_delete::execute (this=0x7f2762266fc0, thd=0x7f27685ac000)
    at /mnt/workspace/percona-server-5.7-binaries-release-rocks/label_exp/centos6-64/percona-server-5.7.19-17/sql/sql_delete.cc:1386
#16 0x0000000000cad230 in mysql_execute_command (thd=0x7f27685ac000, first_level=Unhandled dwarf expression opcode 0xf3)
    at /mnt/workspace/percona-server-5.7-binaries-release-rocks/label_exp/centos6-64/percona-server-5.7.19-17/sql/sql_parse.cc:3758
#17 0x0000000000cb3385 in mysql_parse (thd=0x7f27685ac000, parser_state=Unhandled dwarf expression opcode 0xf3)
    at /mnt/workspace/percona-server-5.7-binaries-release-rocks/label_exp/centos6-64/percona-server-5.7.19-17/sql/sql_parse.cc:5891
#18 0x0000000000cb3eca in dispatch_command (thd=0x7f27685ac000, com_data=Unhandled dwarf expression opcode 0xf3)
    at /mnt/workspace/percona-server-5.7-binaries-release-rocks/label_exp/centos6-64/percona-server-5.7.19-17/sql/sql_parse.cc:1493
#19 0x0000000000cb5917 in do_command (thd=0x7f27685ac000) at /mnt/workspace/percona-server-5.7-binaries-release-rocks/label_exp/centos6-64/percona-server-5.7.19-17/sql/sql_parse.cc:1021
#20 0x0000000000d5a267 in threadpool_process_request (thd=0x7f27685ac000)
    at /mnt/workspace/percona-server-5.7-binaries-release-rocks/label_exp/centos6-64/percona-server-5.7.19-17/sql/threadpool_common.cc:254
#21 0x0000000000d6979e in handle_event (param=0x1e32a00)
    at /mnt/workspace/percona-server-5.7-binaries-release-rocks/label_exp/centos6-64/percona-server-5.7.19-17/sql/threadpool_unix.cc:1578
#22 worker_main (param=0x1e32a00) at /mnt/workspace/percona-server-5.7-binaries-release-rocks/label_exp/centos6-64/percona-server-5.7.19-17/sql/threadpool_unix.cc:1630
#23 0x000000000123cf24 in pfs_spawn_thread (arg=0x7f2743c40620)
    at /mnt/workspace/percona-server-5.7-binaries-release-rocks/label_exp/centos6-64/percona-server-5.7.19-17/storage/perfschema/pfs.cc:2188
#24 0x000000369ea07aa1 in start_thread () from /lib64/libpthread.so.0
#25 0x000000369e2e893d in clone () from /lib64/libc.so.6

How to repeat:
Can't repeat it hasn't reoccurred as of yet.

Suggested fix:
have no idea
[5 May 2022 12:10] MySQL Verification Team
Hi Mr. tu,

Thank you very much for your report.

However, you are using an ancient release of 5.7 that is more then 2 (two) years old. Current release is 5.7.38 and it contains thousands of bugs fixed since 5.7.19.

Let us know if 5.7.38 still causes the problem.