Description:
Hi,
Unfortunately we do not have much good information about the cause. This occurred on our production server at night time, there may have been some cleanup scripts running doing deletes of old records, and optimize table afterwards around the crash.
this database is using about 1.8TB disk, about 800 QPS avg, and nearly 400000 tables. server has 416Gi memory. 12 cores, running in a vmware environment
we where running 8.0.32 (gentoo: dev-db/mysql-8.0.32-r2) at the time of the crash, restarts failed with the same stacktrace. After that we upgraded to latest stable in gentoo, 8.0.37, but with the same crash and stack trace at startup. (see below)
first crash:
2025-03-20T01:04:40.884760Z 0 [ERROR] [MY-013183] [InnoDB] Assertion failure: page0page.ic:704:slot_no != 0 thread 139610756699840
...
2025-03-20T01:04:40Z UTC - mysqld got signal 6 ;
Most likely, you have hit a bug, but this error can also be caused by malfunctioning hardware.
BuildID[sha1]=73043fa2e4359d38545c4842135947c1e55eab31
Thread pointer: 0x7ef565af8cc0
...
stack_bottom = 7ef9a98f4af0 thread_stack 0x100000
/usr/sbin/mysqld(my_print_stacktrace(unsigned char const*, unsigned long)+0x3c) [0x56279b75fd6c]
/usr/sbin/mysqld(print_fatal_signal(int)+0x2c1) [0x56279a4147c1]
/usr/sbin/mysqld(my_server_abort()+0x64) [0x56279a4149d4]
/usr/sbin/mysqld(my_abort()+0xa) [0x56279b759d0a]
/usr/sbin/mysqld(ut_dbg_assertion_failed(char const*, char const*, unsigned long)+0x2ec) [0x56279ba0de4c]
/usr/sbin/mysqld(rtr_page_copy_rec_list_end_no_locks(buf_block_t*, buf_block_t*, unsigned char*, dict_index_t*, mem_block_info_t*, rtr_rec_move*, unsigned long, uns
igned long*, mtr_t*)+0xcbf) [0x56279bc0cb7f]
/usr/sbin/mysqld(page_copy_rec_list_end(buf_block_t*, buf_block_t*, unsigned char*, dict_index_t*, mtr_t*)+0x2fd) [0x56279b8ef3ad]
/usr/sbin/mysqld(btr_compress(btr_cur_t*, bool, mtr_t*)+0xaf7) [0x56279ba4b3b7]
/usr/sbin/mysqld(btr_cur_pessimistic_delete(dberr_t*, bool, btr_cur_t*, unsigned int, bool, unsigned long, unsigned long, unsigned long, mtr_t*, btr_pcur_t*, purge_
node_t*)+0x226) [0x56279ba5e926]
/usr/sbin/mysqld(rtr_node_ptr_delete(btr_cur_t*, mtr_t*)+0x46) [0x56279bc0ad86]
/usr/sbin/mysqld(rtr_merge_and_update_mbr(btr_cur_t*, btr_cur_t*, unsigned long*, unsigned long*, unsigned char*, mtr_t*)+0x93) [0x56279bc0ae83]
/usr/sbin/mysqld(btr_compress(btr_cur_t*, bool, mtr_t*)+0xc2e) [0x56279ba4b4ee]
/usr/sbin/mysqld(btr_cur_pessimistic_delete(dberr_t*, bool, btr_cur_t*, unsigned int, bool, unsigned long, unsigned long, unsigned long, mtr_t*, btr_pcur_t*, purge_
node_t*)+0x226) [0x56279ba5e926]
/usr/sbin/mysqld(rtr_node_ptr_delete(btr_cur_t*, mtr_t*)+0x46) [0x56279bc0ad86]
/usr/sbin/mysqld(rtr_merge_and_update_mbr(btr_cur_t*, btr_cur_t*, unsigned long*, unsigned long*, unsigned char*, mtr_t*)+0x93) [0x56279bc0ae83]
/usr/sbin/mysqld(btr_compress(btr_cur_t*, bool, mtr_t*)+0x1881) [0x56279ba4c141]
/usr/sbin/mysqld(btr_cur_pessimistic_delete(dberr_t*, bool, btr_cur_t*, unsigned int, bool, unsigned long, unsigned long, unsigned long, mtr_t*, btr_pcur_t*, purge_
node_t*)+0x226) [0x56279ba5e926]
/usr/sbin/mysqld(+0x25779e8) [0x56279b95c9e8]
/usr/sbin/mysqld(row_purge_step(que_thr_t*)+0x1017) [0x56279b95f537]
/usr/sbin/mysqld(que_run_threads(que_thr_t*)+0x89b) [0x56279b90af7b]
/usr/sbin/mysqld(srv_worker_thread()+0x6a4) [0x56279b998494]
/usr/sbin/mysqld(std::thread::_State_impl<std::thread::_Invoker<std::tuple<Detached_thread, void (*)()> > >::_M_run()+0xd5) [0x56279b8a0b65]
/usr/lib/gcc/x86_64-pc-linux-gnu/13/libstdc++.so.6(+0xf2fcf) [0x7f02a72f2fcf]
/lib64/libc.so.6(+0x8293b) [0x7f02a6fd193b]
/lib64/libc.so.6(+0x1027c8) [0x7f02a70517c8]
...
restarts after that resulted in this similar stacktraces at startup
After upgrade to 8.0.37:
2025-03-20T02:08:51.496629Z 0 [ERROR] [MY-013183] [InnoDB] Assertion failure: page0page.ic:705:slot_no != 0 thread 139624336840384
...
2025-03-20T02:08:51Z UTC - mysqld got signal 6 ;
Most likely, you have hit a bug, but this error can also be caused by malfunctioning hardware.
Thread pointer: 0x7ed991d17450
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 7efcd2ffca10 thread_stack 0x100000
...
/usr/sbin/mysqld(ut_dbg_assertion_failed(char const*, char const*, unsigned long)+0x2ec) [0x55818dd46a4c]
/usr/sbin/mysqld(rtr_page_copy_rec_list_end_no_locks(buf_block_t*, buf_block_t*, unsigned char*, dict_index_t*, mem_block_info_t*, rtr_rec_move*, unsigned long, unsigned long*, mtr_t*)+0xcbf) [0x55818df4a7df]
/usr/sbin/mysqld(page_copy_rec_list_end(buf_block_t*, buf_block_t*, unsigned char*, dict_index_t*, mtr_t*)+0x2fd) [0x55818dc20ddd]
/usr/sbin/mysqld(btr_compress(btr_cur_t*, bool, mtr_t*)+0xaf7) [0x55818dd850f7]
/usr/sbin/mysqld(btr_cur_pessimistic_delete(dberr_t*, bool, btr_cur_t*, unsigned int, bool, unsigned long, unsigned long, unsigned long, mtr_t*, btr_pcur_t*, purge_node_t*)+0x226) [0x55818dd98916]
/usr/sbin/mysqld(rtr_node_ptr_delete(btr_cur_t*, mtr_t*)+0x46) [0x55818df489e6]
/usr/sbin/mysqld(rtr_merge_and_update_mbr(btr_cur_t*, btr_cur_t*, unsigned long*, unsigned long*, unsigned char*, mtr_t*)+0x93) [0x55818df48ae3]
/usr/sbin/mysqld(btr_compress(btr_cur_t*, bool, mtr_t*)+0xc2e) [0x55818dd8522e]
/usr/sbin/mysqld(btr_cur_pessimistic_delete(dberr_t*, bool, btr_cur_t*, unsigned int, bool, unsigned long, unsigned long, unsigned long, mtr_t*, btr_pcur_t*, purge_node_t*)+0x226) [0x55818dd98916]
/usr/sbin/mysqld(rtr_node_ptr_delete(btr_cur_t*, mtr_t*)+0x46) [0x55818df489e6]
/usr/sbin/mysqld(rtr_merge_and_update_mbr(btr_cur_t*, btr_cur_t*, unsigned long*, unsigned long*, unsigned char*, mtr_t*)+0x93) [0x55818df48ae3]
/usr/sbin/mysqld(btr_compress(btr_cur_t*, bool, mtr_t*)+0x1881) [0x55818dd85e81]
/usr/sbin/mysqld(btr_cur_pessimistic_delete(dberr_t*, bool, btr_cur_t*, unsigned int, bool, unsigned long, unsigned long, unsigned long, mtr_t*, btr_pcur_t*, purge_
node_t*)+0x226) [0x55818dd98916]
/usr/sbin/mysqld(+0x209f8d8) [0x55818dc908d8]
/usr/sbin/mysqld(row_purge_step(que_thr_t*)+0x107e) [0x55818dc934ce]
/usr/sbin/mysqld(que_run_threads(que_thr_t*)+0x918) [0x55818dc3df98]
/usr/sbin/mysqld(trx_purge(unsigned long, unsigned long, bool)+0x385) [0x55818dd02935]
/usr/sbin/mysqld(srv_purge_coordinator_thread()+0x4c4) [0x55818dcce944]
/usr/sbin/mysqld(std::thread::_State_impl<std::thread::_Invoker<std::tuple<Detached_thread, void (*)()> > >::_M_run()+0xd5) [0x55818dbd1c55]
/usr/lib/gcc/x86_64-pc-linux-gnu/13/libstdc++.so.6(+0xf2fcf) [0x7f112c8f2fcf]
/lib64/libc.so.6(+0x8293b) [0x7f112c5d193b]
/lib64/libc.so.6(+0x1027c8) [0x7f112c6517c8]
For both versions;
innodb_force_recovery = 1 did not help, same problem, but
innodb_force_recovery = 2 allowed us to start in read only mode.
after some tries after that, still getting the crash with innodb_force_recovery = 1 or 0 , but suddenly, it started up ok in readwrite mode (0).
When it did start up ok, I just had removed these two lines:
log-replica-updates = ON
replica_parallel_workers = 4
Explanation: this instance has earlier acted as a replica, but was promoted to a source a year ago, so those config lines remained, I'm not convinced it was those config lines that caused the crash at startup though...
Additional information, an AI analyzis (kagi.com) of the scenario and crash dump suggests to check our Spatial indexes ("Since the error occurs in R-tree functions"), and that is actually quite plausible, since we use Spatial indexes and MBR queries quite extensively. (linestring)
Question; if this occurs again, how can we exctract a bit more information about the cause, to help debugging? I did not seem to find anything usefull in the innodb status variables when running in readonly.
kind regards,
Håkon Innerdal
How to repeat:
Unknown
Suggested fix:
Unknown
Description: Hi, Unfortunately we do not have much good information about the cause. This occurred on our production server at night time, there may have been some cleanup scripts running doing deletes of old records, and optimize table afterwards around the crash. this database is using about 1.8TB disk, about 800 QPS avg, and nearly 400000 tables. server has 416Gi memory. 12 cores, running in a vmware environment we where running 8.0.32 (gentoo: dev-db/mysql-8.0.32-r2) at the time of the crash, restarts failed with the same stacktrace. After that we upgraded to latest stable in gentoo, 8.0.37, but with the same crash and stack trace at startup. (see below) first crash: 2025-03-20T01:04:40.884760Z 0 [ERROR] [MY-013183] [InnoDB] Assertion failure: page0page.ic:704:slot_no != 0 thread 139610756699840 ... 2025-03-20T01:04:40Z UTC - mysqld got signal 6 ; Most likely, you have hit a bug, but this error can also be caused by malfunctioning hardware. BuildID[sha1]=73043fa2e4359d38545c4842135947c1e55eab31 Thread pointer: 0x7ef565af8cc0 ... stack_bottom = 7ef9a98f4af0 thread_stack 0x100000 /usr/sbin/mysqld(my_print_stacktrace(unsigned char const*, unsigned long)+0x3c) [0x56279b75fd6c] /usr/sbin/mysqld(print_fatal_signal(int)+0x2c1) [0x56279a4147c1] /usr/sbin/mysqld(my_server_abort()+0x64) [0x56279a4149d4] /usr/sbin/mysqld(my_abort()+0xa) [0x56279b759d0a] /usr/sbin/mysqld(ut_dbg_assertion_failed(char const*, char const*, unsigned long)+0x2ec) [0x56279ba0de4c] /usr/sbin/mysqld(rtr_page_copy_rec_list_end_no_locks(buf_block_t*, buf_block_t*, unsigned char*, dict_index_t*, mem_block_info_t*, rtr_rec_move*, unsigned long, uns igned long*, mtr_t*)+0xcbf) [0x56279bc0cb7f] /usr/sbin/mysqld(page_copy_rec_list_end(buf_block_t*, buf_block_t*, unsigned char*, dict_index_t*, mtr_t*)+0x2fd) [0x56279b8ef3ad] /usr/sbin/mysqld(btr_compress(btr_cur_t*, bool, mtr_t*)+0xaf7) [0x56279ba4b3b7] /usr/sbin/mysqld(btr_cur_pessimistic_delete(dberr_t*, bool, btr_cur_t*, unsigned int, bool, unsigned long, unsigned long, unsigned long, mtr_t*, btr_pcur_t*, purge_ node_t*)+0x226) [0x56279ba5e926] /usr/sbin/mysqld(rtr_node_ptr_delete(btr_cur_t*, mtr_t*)+0x46) [0x56279bc0ad86] /usr/sbin/mysqld(rtr_merge_and_update_mbr(btr_cur_t*, btr_cur_t*, unsigned long*, unsigned long*, unsigned char*, mtr_t*)+0x93) [0x56279bc0ae83] /usr/sbin/mysqld(btr_compress(btr_cur_t*, bool, mtr_t*)+0xc2e) [0x56279ba4b4ee] /usr/sbin/mysqld(btr_cur_pessimistic_delete(dberr_t*, bool, btr_cur_t*, unsigned int, bool, unsigned long, unsigned long, unsigned long, mtr_t*, btr_pcur_t*, purge_ node_t*)+0x226) [0x56279ba5e926] /usr/sbin/mysqld(rtr_node_ptr_delete(btr_cur_t*, mtr_t*)+0x46) [0x56279bc0ad86] /usr/sbin/mysqld(rtr_merge_and_update_mbr(btr_cur_t*, btr_cur_t*, unsigned long*, unsigned long*, unsigned char*, mtr_t*)+0x93) [0x56279bc0ae83] /usr/sbin/mysqld(btr_compress(btr_cur_t*, bool, mtr_t*)+0x1881) [0x56279ba4c141] /usr/sbin/mysqld(btr_cur_pessimistic_delete(dberr_t*, bool, btr_cur_t*, unsigned int, bool, unsigned long, unsigned long, unsigned long, mtr_t*, btr_pcur_t*, purge_ node_t*)+0x226) [0x56279ba5e926] /usr/sbin/mysqld(+0x25779e8) [0x56279b95c9e8] /usr/sbin/mysqld(row_purge_step(que_thr_t*)+0x1017) [0x56279b95f537] /usr/sbin/mysqld(que_run_threads(que_thr_t*)+0x89b) [0x56279b90af7b] /usr/sbin/mysqld(srv_worker_thread()+0x6a4) [0x56279b998494] /usr/sbin/mysqld(std::thread::_State_impl<std::thread::_Invoker<std::tuple<Detached_thread, void (*)()> > >::_M_run()+0xd5) [0x56279b8a0b65] /usr/lib/gcc/x86_64-pc-linux-gnu/13/libstdc++.so.6(+0xf2fcf) [0x7f02a72f2fcf] /lib64/libc.so.6(+0x8293b) [0x7f02a6fd193b] /lib64/libc.so.6(+0x1027c8) [0x7f02a70517c8] ... restarts after that resulted in this similar stacktraces at startup After upgrade to 8.0.37: 2025-03-20T02:08:51.496629Z 0 [ERROR] [MY-013183] [InnoDB] Assertion failure: page0page.ic:705:slot_no != 0 thread 139624336840384 ... 2025-03-20T02:08:51Z UTC - mysqld got signal 6 ; Most likely, you have hit a bug, but this error can also be caused by malfunctioning hardware. Thread pointer: 0x7ed991d17450 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 7efcd2ffca10 thread_stack 0x100000 ... /usr/sbin/mysqld(ut_dbg_assertion_failed(char const*, char const*, unsigned long)+0x2ec) [0x55818dd46a4c] /usr/sbin/mysqld(rtr_page_copy_rec_list_end_no_locks(buf_block_t*, buf_block_t*, unsigned char*, dict_index_t*, mem_block_info_t*, rtr_rec_move*, unsigned long, unsigned long*, mtr_t*)+0xcbf) [0x55818df4a7df] /usr/sbin/mysqld(page_copy_rec_list_end(buf_block_t*, buf_block_t*, unsigned char*, dict_index_t*, mtr_t*)+0x2fd) [0x55818dc20ddd] /usr/sbin/mysqld(btr_compress(btr_cur_t*, bool, mtr_t*)+0xaf7) [0x55818dd850f7] /usr/sbin/mysqld(btr_cur_pessimistic_delete(dberr_t*, bool, btr_cur_t*, unsigned int, bool, unsigned long, unsigned long, unsigned long, mtr_t*, btr_pcur_t*, purge_node_t*)+0x226) [0x55818dd98916] /usr/sbin/mysqld(rtr_node_ptr_delete(btr_cur_t*, mtr_t*)+0x46) [0x55818df489e6] /usr/sbin/mysqld(rtr_merge_and_update_mbr(btr_cur_t*, btr_cur_t*, unsigned long*, unsigned long*, unsigned char*, mtr_t*)+0x93) [0x55818df48ae3] /usr/sbin/mysqld(btr_compress(btr_cur_t*, bool, mtr_t*)+0xc2e) [0x55818dd8522e] /usr/sbin/mysqld(btr_cur_pessimistic_delete(dberr_t*, bool, btr_cur_t*, unsigned int, bool, unsigned long, unsigned long, unsigned long, mtr_t*, btr_pcur_t*, purge_node_t*)+0x226) [0x55818dd98916] /usr/sbin/mysqld(rtr_node_ptr_delete(btr_cur_t*, mtr_t*)+0x46) [0x55818df489e6] /usr/sbin/mysqld(rtr_merge_and_update_mbr(btr_cur_t*, btr_cur_t*, unsigned long*, unsigned long*, unsigned char*, mtr_t*)+0x93) [0x55818df48ae3] /usr/sbin/mysqld(btr_compress(btr_cur_t*, bool, mtr_t*)+0x1881) [0x55818dd85e81] /usr/sbin/mysqld(btr_cur_pessimistic_delete(dberr_t*, bool, btr_cur_t*, unsigned int, bool, unsigned long, unsigned long, unsigned long, mtr_t*, btr_pcur_t*, purge_ node_t*)+0x226) [0x55818dd98916] /usr/sbin/mysqld(+0x209f8d8) [0x55818dc908d8] /usr/sbin/mysqld(row_purge_step(que_thr_t*)+0x107e) [0x55818dc934ce] /usr/sbin/mysqld(que_run_threads(que_thr_t*)+0x918) [0x55818dc3df98] /usr/sbin/mysqld(trx_purge(unsigned long, unsigned long, bool)+0x385) [0x55818dd02935] /usr/sbin/mysqld(srv_purge_coordinator_thread()+0x4c4) [0x55818dcce944] /usr/sbin/mysqld(std::thread::_State_impl<std::thread::_Invoker<std::tuple<Detached_thread, void (*)()> > >::_M_run()+0xd5) [0x55818dbd1c55] /usr/lib/gcc/x86_64-pc-linux-gnu/13/libstdc++.so.6(+0xf2fcf) [0x7f112c8f2fcf] /lib64/libc.so.6(+0x8293b) [0x7f112c5d193b] /lib64/libc.so.6(+0x1027c8) [0x7f112c6517c8] For both versions; innodb_force_recovery = 1 did not help, same problem, but innodb_force_recovery = 2 allowed us to start in read only mode. after some tries after that, still getting the crash with innodb_force_recovery = 1 or 0 , but suddenly, it started up ok in readwrite mode (0). When it did start up ok, I just had removed these two lines: log-replica-updates = ON replica_parallel_workers = 4 Explanation: this instance has earlier acted as a replica, but was promoted to a source a year ago, so those config lines remained, I'm not convinced it was those config lines that caused the crash at startup though... Additional information, an AI analyzis (kagi.com) of the scenario and crash dump suggests to check our Spatial indexes ("Since the error occurs in R-tree functions"), and that is actually quite plausible, since we use Spatial indexes and MBR queries quite extensively. (linestring) Question; if this occurs again, how can we exctract a bit more information about the cause, to help debugging? I did not seem to find anything usefull in the innodb status variables when running in readonly. kind regards, Håkon Innerdal How to repeat: Unknown Suggested fix: Unknown