Again faced the same issue on another node, seems like we are getting hit by this bug whenever it's trying to access "role" table either Insert or Update and it's not every time, We see many DML's passes for the same query but once in a blue moon it crashes the MySQL. fyi, "role" is our internal table 2023-06-29T00:22:33.395669Z 1205 [ERROR] [MY-012153] [InnoDB] Trying to access page number 1315925605 in space 4294967278, space name innodb_undo_002, which is outside the tablespace bounds. Byte offset 0 , len 16384, i/o type read. If you get this error at mysqld startup, please check that your my.cnf matches the ibdata files that you have in the MySQL server. 2023-06-29T00:22:33.395704Z 1205 [ERROR] [MY-012154] [InnoDB] Server exits. 2023-06-29T00:22:33.395726Z 1205 [ERROR] [MY-013183] [InnoDB] Assertion failure: fil0fil.cc:7547 thread 140132859459328 InnoDB: We intentionally generate a memory trap. InnoDB: Submit a detailed bug report to http://bugs.mysql.com. InnoDB: If you get repeated assertion failures or crashes, even InnoDB: immediately after the mysqld startup, there may be InnoDB: corruption in the InnoDB tablespace. Please refer to InnoDB: http://dev.mysql.com/doc/refman/8.0/en/forcing-innodb-recovery.html InnoDB: about forcing recovery. 2023-06-29T00:22:33Z UTC - mysqld got signal 6 ; Most likely, you have hit a bug, but this error can also be caused by malfunctioning hardware. BuildID[sha1]=f5a4b8541445e3f1cd10798a1d9f0ca415d73180 Server Version: 8.0.32-24 Percona Server (GPL), Release 24, Revision e5c6e9d2 Thread pointer: 0x7f72cd01e000 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 7f73394f2af0 thread_stack 0x40000 /usr/sbin/mysqld(my_print_stacktrace(unsigned char const*, unsigned long)+0x3d) [0x2153fed] /usr/sbin/mysqld(print_fatal_signal(int)+0x39f) [0x11f020f] /usr/sbin/mysqld(my_server_abort()+0x7e) [0x11f03be] /usr/sbin/mysqld(my_abort()+0xa) [0x214dfaa] /usr/sbin/mysqld(ut_dbg_assertion_failed(char const*, char const*, unsigned long)+0x31f) [0x24158cf] /usr/sbin/mysqld() [0x254f796] /usr/sbin/mysqld(Fil_shard::do_io(IORequest const&, bool, page_id_t const&, page_size_t const&, unsigned long, unsigned long, void*, void*, trx_t*, bool)+0xd63) [0x2569a43] /usr/sbin/mysqld(_fil_io(IORequest const&, bool, page_id_t const&, page_size_t const&, unsigned long, unsigned long, void*, void*, trx_t*, bool)+0x82) [0x2569b02] /usr/sbin/mysqld(buf_read_page_low(dberr_t*, bool, unsigned long, unsigned long, page_id_t const&, page_size_t const&, bool, trx_t*, bool)+0x184) [0x249e8e4] /usr/sbin/mysqld(buf_read_page(page_id_t const&, page_size_t const&, trx_t*)+0x49) [0x249edd9] /usr/sbin/mysqld(Buf_fetch::read_page()+0x38) [0x246b598] /usr/sbin/mysqld(Buf_fetch_normal::get(buf_block_t*&)+0x170) [0x24771d0] /usr/sbin/mysqld(Buf_fetch::single_page()+0x51) [0x24772b1] /usr/sbin/mysqld(buf_page_get_gen(page_id_t const&, page_size_t const&, unsigned long, buf_block_t*, Page_fetch, ut::Location, mtr_t*, bool)+0x2f9) [0x2478229] /usr/sbin/mysqld() [0x23f09f6] /usr/sbin/mysqld(trx_undo_prev_version_build(unsigned char const*, mtr_t*, unsigned char const*, dict_index_t const*, unsigned long*, mem_block_info_t*, unsigned char**, mem_block_info_t*, dtuple_t const* *, unsigned long, lob::undo_vers_t*)+0x206) [0x23f37b6] /usr/sbin/mysqld(row_vers_impl_x_locked(unsigned char const*, dict_index_t const*, unsigned long const*)+0x377) [0x23b4897] /usr/sbin/mysqld(lock_rec_convert_impl_to_expl(buf_block_t const*, unsigned char const*, dict_index_t*, unsigned long const*)+0x337) [0x22c9397] /usr/sbin/mysqld(lock_sec_rec_read_check_and_lock(lock_duration_t, buf_block_t const*, unsigned char const*, dict_index_t*, unsigned long const*, select_mode, lock_mode, unsigned long, que_thr_t*)+0x1c6) [0x22c99d6] /usr/sbin/mysqld() [0x2361845] /usr/sbin/mysqld(row_ins_sec_index_entry_low(unsigned int, unsigned long, dict_index_t*, mem_block_info_t*, mem_block_info_t*, dtuple_t*, unsigned long, que_thr_t*, bool)+0xf9d) [0x2365aed] /usr/sbin/mysqld(row_ins_sec_index_entry(dict_index_t*, dtuple_t*, que_thr_t*, bool)+0x16a) [0x2369bba] /usr/sbin/mysqld() [0x23abe05] /usr/sbin/mysqld(row_upd_step(que_thr_t*)+0x63d) [0x23acc8d] /usr/sbin/mysqld() [0x23799d1] /usr/sbin/mysqld(ha_innobase::update_row(unsigned char const*, unsigned char*)+0x3a0) [0x2260e80] /usr/sbin/mysqld(handler::ha_update_row(unsigned char const*, unsigned char*)+0x213) [0xd6f393] /usr/sbin/mysqld(Sql_cmd_update::update_single_table(THD*)+0x217f) [0x115a34f] /usr/sbin/mysqld(Sql_cmd_dml::execute(THD*)+0x189) [0x10c86b9] /usr/sbin/mysqld(mysql_execute_command(THD*, bool)+0xb46) [0x106a9b6] /usr/sbin/mysqld(dispatch_sql_command(THD*, Parser_state*, bool)+0x5c0) [0x106e9e0] /usr/sbin/mysqld(dispatch_command(THD*, COM_DATA const*, enum_server_command)+0x21b4) [0x1071154] /usr/sbin/mysqld(do_command(THD*)+0x282) [0x1072032] /usr/sbin/mysqld() [0x11e01a0] /usr/sbin/mysqld() [0x2620705] /lib64/libpthread.so.0(+0x7ea5) [0x7f759454dea5] /lib64/libc.so.6(clone+0x6d) [0x7f7592908b0d] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (7f7294479030): update role set name = 'None', pumber = 110234, next_date = now(), role_type = null, manage_power = 0, manage_standard = 0 where role_id = 'XXXX-XXXX-XXXX-XXXX-XXXX' and idx = 'XXXX-XXXXX-XXXX-XXXX-XXXX' /* RT-XXXXXX:XXXXXXXX:XXXX-XXX*/ /*DML*/ Connection ID (thread ID): 1205 Status: NOT_KILLED Please help us make Percona Server better by reporting any bugs at https://bugs.percona.com/ You may download the Percona Server operations manual by visiting http://www.percona.com/software/percona-server/. You may find information in the manual which will help you identify the cause of the crash. 2023-06-29T00:22:37.253061Z 0 [Warning] [MY-011068] [Server] The syntax 'skip_slave_start' is deprecated and will be removed in a future release. Please use skip_replica_start instead. 2023-06-29T00:22:37.255206Z 0 [Warning] [MY-010097] [Server] Insecure configuration for --secure-log-path: Current value does not restrict location of generated files. Consider setting it to a valid, non- empty path. 2023-06-29T00:22:37.255268Z 0 [System] [MY-010116] [Server] /usr/sbin/mysqld (mysqld 8.0.32-24) starting as process 7251 2023-06-29T00:22:37.278686Z 0 [Warning] [MY-013907] [InnoDB] Deprecated configuration parameters innodb_log_file_size and/or innodb_log_files_in_group have been used to compute innodb_redo_log_capacity=21 47483648. Please use innodb_redo_log_capacity instead. 2023-06-29T00:22:37.284206Z 1 [System] [MY-013576] [InnoDB] InnoDB initialization has started. 2023-06-29T00:22:50.614266Z 0 [ERROR] [MY-013183] [InnoDB] Assertion failure: rec.cc:384 thread 140241792923392 InnoDB: We intentionally generate a memory trap. InnoDB: Submit a detailed bug report to http://bugs.mysql.com. InnoDB: If you get repeated assertion failures or crashes, even InnoDB: immediately after the mysqld startup, there may be InnoDB: corruption in the InnoDB tablespace. Please refer to InnoDB: http://dev.mysql.com/doc/refman/8.0/en/forcing-innodb-recovery.html InnoDB: about forcing recovery. 2023-06-29T00:22:50Z UTC - mysqld got signal 6 ; Most likely, you have hit a bug, but this error can also be caused by malfunctioning hardware. BuildID[sha1]=f5a4b8541445e3f1cd10798a1d9f0ca415d73180 Server Version: 8.0.32-24 Percona Server (GPL), Release 24, Revision e5c6e9d2 Thread pointer: 0x0 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0 thread_stack 0x40000 /usr/sbin/mysqld(my_print_stacktrace(unsigned char const*, unsigned long)+0x3d) [0x2153fed] /usr/sbin/mysqld(print_fatal_signal(int)+0x39f) [0x11f020f] /usr/sbin/mysqld(my_server_abort()+0x7e) [0x11f03be] /usr/sbin/mysqld(my_abort()+0xa) [0x214dfaa] /usr/sbin/mysqld(ut_dbg_assertion_failed(char const*, char const*, unsigned long)+0x31f) [0x24158cf] /usr/sbin/mysqld(rec_get_offsets(unsigned char const*, dict_index_t const*, unsigned long*, unsigned long, ut::Location, mem_block_info_t**)+0xb7) [0x2423107] /usr/sbin/mysqld(page_cur_parse_delete_rec(unsigned char*, unsigned char*, buf_block_t*, dict_index_t*, mtr_t*)+0xdf) [0x2339b0f] /usr/sbin/mysqld() [0x23099ce] /usr/sbin/mysqld(recv_recover_page_func(bool, buf_block_t*)+0x64b) [0x230c46b] /usr/sbin/mysqld(buf_page_io_complete(buf_page_t*, bool)+0x4a0) [0x2479140] /usr/sbin/mysqld(fil_aio_wait(unsigned long)+0x18e) [0x2562c4e] /usr/sbin/mysqld() [0x23c7218] /usr/sbin/mysqld(std::thread::_State_impl > >::_M_run()+0xae) [0x23c7a2e] /usr/sbin/mysqld() [0x2b367d4] /lib64/libpthread.so.0(+0x7ea5) [0x7f8cafc05ea5] /lib64/libc.so.6(clone+0x6d) [0x7f8cadfc0b0d] Please help us make Percona Server better by reporting any bugs at https://bugs.percona.com/ You may download the Percona Server operations manual by visiting http://www.percona.com/software/percona-server/. You may find information in the manual which will help you identify the cause of the crash. 2023-06-29T00:22:53.000001Z 0 [Warning] [MY-011068] [Server] The syntax 'skip_slave_start' is deprecated and will be removed in a future release. Please use skip_replica_start instead. 2023-06-29T00:22:53.001945Z 0 [Warning] [MY-010097] [Server] Insecure configuration for --secure-log-path: Current value does not restrict location of generated files. Consider setting it to a valid, non- empty path. 2023-06-29T00:22:53.002008Z 0 [System] [MY-010116] [Server] /usr/sbin/mysqld (mysqld 8.0.32-24) starting as process 7332 2023-06-29T00:22:53.023836Z 0 [Warning] [MY-013907] [InnoDB] Deprecated configuration parameters innodb_log_file_size and/or innodb_log_files_in_group have been used to compute innodb_redo_log_capacity=21 47483648. Please use innodb_redo_log_capacity instead. 2023-06-29T00:22:53.026971Z 1 [System] [MY-013576] [InnoDB] InnoDB initialization has started. 2023-06-29T00:23:02.520146Z 0 [ERROR] [MY-013183] [InnoDB] Assertion failure: rec.cc:384 thread 139732306609920 InnoDB: We intentionally generate a memory trap. InnoDB: Submit a detailed bug report to http://bugs.mysql.com. InnoDB: If you get repeated assertion failures or crashes, even InnoDB: immediately after the mysqld startup, there may be InnoDB: corruption in the InnoDB tablespace. Please refer to InnoDB: http://dev.mysql.com/doc/refman/8.0/en/forcing-innodb-recovery.html InnoDB: about forcing recovery. 2023-06-29T00:23:02Z UTC - mysqld got signal 6 ; Most likely, you have hit a bug, but this error can also be caused by malfunctioning hardware. BuildID[sha1]=f5a4b8541445e3f1cd10798a1d9f0ca415d73180 Server Version: 8.0.32-24 Percona Server (GPL), Release 24, Revision e5c6e9d2 mysql> show create table role\G *************************** 1. row *************************** Table: role Create Table: CREATE TABLE `role` ( `role_id` char(36) COLLATE utf8mb4_bin NOT NULL, `workgroup_id` char(36) COLLATE utf8mb4_bin NOT NULL, `name` varchar(40) COLLATE utf8mb4_bin NOT NULL, `partition_number` int unsigned NOT NULL, `can_manage_power_users` tinyint(1) NOT NULL DEFAULT '0', `can_manage_standard_users` tinyint(1) NOT NULL DEFAULT '0', `can_manage_groups` tinyint(1) NOT NULL DEFAULT '0', `can_manage_billing` tinyint(1) NOT NULL DEFAULT '0', `can_manage_account_conf` tinyint(1) NOT NULL DEFAULT '0', `can_access_reports` tinyint(1) NOT NULL DEFAULT '0', `can_run_reports_with_all_files` tinyint(1) DEFAULT '0', `can_manage_personal_dev` tinyint(1) NOT NULL DEFAULT '0', `can_manage_storage_dev` tinyint(1) NOT NULL DEFAULT '0', `can_view_trash` tinyint(1) NOT NULL DEFAULT '0', `can_create_top_folders` tinyint(1) NOT NULL DEFAULT '0', `can_use_offline_sync` tinyint(1) NOT NULL DEFAULT '0', `can_manage_metadata_settings` tinyint(1) NOT NULL DEFAULT '0', `can_manage_etmf` tinyint(1) NOT NULL DEFAULT '0', `can_manage_links` tinyint(1) NOT NULL DEFAULT '0', `can_manage_projects` tinyint(1) NOT NULL DEFAULT '0', `can_manage_document_rooms` tinyint(1) NOT NULL DEFAULT '0', `can_manage_workflow_templates` tinyint NOT NULL DEFAULT '0', `can_access_gxp_portal` tinyint NOT NULL DEFAULT '0', `can_manage_quality_docs` tinyint(1) NOT NULL DEFAULT '0', `creation_date` datetime NOT NULL, `update_date` datetime DEFAULT NULL, `free_role_type` enum('MSP') COLLATE utf8mb4_bin DEFAULT NULL COMMENT 'roles with non null free_account_type values are not counted towards role limits', PRIMARY KEY (`role_id`,`workgroup_id`), UNIQUE KEY `role_name_uq` (`name`,`workgroup_id`), KEY `role_i1` (`workgroup_id`) ) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_bin;