| Bug #108086 | Backup crash server - Assertion failure: block->page.get_space() != nullptr | ||
|---|---|---|---|
| Submitted: | 8 Aug 2022 5:39 | Modified: | 18 Nov 2022 4:45 |
| Reporter: | Johannes Pretorius | Email Updates: | |
| Status: | Can't repeat | Impact on me: | |
| Category: | MySQL Server | Severity: | S2 (Serious) |
| Version: | Server version: 8.0.30-0ubuntu0.20.04.2 | OS: | Ubuntu (20) |
| Assigned to: | CPU Architecture: | Any | |
| Tags: | assertion failure, Backup, page.get_space | ||
[8 Aug 2022 5:39]
Johannes Pretorius
[8 Aug 2022 11:42]
MySQL Verification Team
Hi Mr. Pretorius, Thank you for your bug report. However, we have received and verified a number of bug reports that occur under completely different conditions. Some are caused by too low buffer pool, some by tablespace recovery, but your new report is something completely different. Hence, what we need is a repeatable test case in the form of the set of SQL statements that always lead to this corruption. If you can not repeat it, then this could just be a temporary glitch in the OS or hardware, which are untraceable. However, if you can provide us with a repeatable test case, then we would denote it as a Security Vulnerability, which would then put this report on the faster track. Please read further on regarding how to prepare a describe report for us ...... Not enough information was provided for us to be able to handle this bug. Please re-read the instructions at http://bugs.mysql.com/how-to-report.php If you can provide more information, feel free to add it to this bug and we will change the status back to 'Open'. Thank you for your interest in MySQL.
[17 Nov 2022 19:53]
Andreas Wachowski
To add a data point, we have seen this problem as well, twice so far, four weeks apart, on two Percona XtraDB clusters (one development, one production), with the following stack trace. It happened during normal operations, as a result, the cluster degraded from three to two instances (AWS EC2 with Debian Bullseye). The relevant excerpt from the error log is: 2022-11-16T15:21:25.836757Z 811016 [ERROR] [MY-013183] [InnoDB] Assertion failure: buf0buf.cc:3276:block->page.get_space() != nullptr thread 139711265851136 InnoDB: We intentionally generate a memory trap. InnoDB: Submit a detailed bug report to http://bugs.mysql.com. InnoDB: If you get repeated assertion failures or crashes, even InnoDB: immediately after the mysqld startup, there may be InnoDB: corruption in the InnoDB tablespace. Please refer to InnoDB: http://dev.mysql.com/doc/refman/8.0/en/forcing-innodb-recovery.html InnoDB: about forcing recovery. 15:21:25 UTC - mysqld got signal 6 ; Most likely, you have hit a bug, but this error can also be caused by malfunctioning hardware. Build ID: 2019d4293e5ff25b16aa2e2bb44132932544d4b1 Server Version: 8.0.29-21.1 Percona XtraDB Cluster (GPL), Release rel21, Revision 250bc93, WSREP version 26.4.3, wsrep_26.4.3 Thread pointer: 0x7f110820e270 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 7f11105f3d20 thread_stack 0x100000 /usr/sbin/mysqld(my_print_stacktrace(unsigned char const*, unsigned long)+0x2e) [0x55d2b61a356e] /usr/sbin/mysqld(print_fatal_signal(int)+0x313) [0x55d2b543cc73] /usr/sbin/mysqld(my_server_abort()+0x5e) [0x55d2b543ce0e] /usr/sbin/mysqld(my_abort()+0xa) [0x55d2b619d76a] /usr/sbin/mysqld(ut_dbg_assertion_failed(char const*, char const*, unsigned long)+0x327) [0x55d2b64891d7] /usr/sbin/mysqld(+0x24d0b1c) [0x55d2b64e2b1c] /usr/sbin/mysqld(+0x24d6d1b) [0x55d2b64e8d1b] /usr/sbin/mysqld(buf_page_init_for_read(dberr_t*, unsigned long, page_id_t const&, page_size_t const&, bool)+0x28d) [0x55d2b64ed1ed] /usr/sbin/mysqld(buf_read_page_low(dberr_t*, bool, unsigned long, unsigned long, page_id_t const&, page_size_t const&, bool, trx_t*, bool)+0x98) [0x55d2b651cdf8] /usr/sbin/mysqld(buf_read_page(page_id_t const&, page_size_t const&, trx_t*)+0x3a) [0x55d2b651d52a] /usr/sbin/mysqld(Buf_fetch<Buf_fetch_normal>::read_page()+0x2a) [0x55d2b64e909a] /usr/sbin/mysqld(Buf_fetch_normal::get(buf_block_t*&)+0x68) [0x55d2b64ea6b8] /usr/sbin/mysqld(Buf_fetch<Buf_fetch_normal>::single_page()+0x3b) [0x55d2b64f2e2b] /usr/sbin/mysqld(buf_page_get_gen(page_id_t const&, page_size_t const&, unsigned long, buf_block_t*, Page_fetch, ut::Location, mtr_t*, bool)+0x215) [0x55d2b64f39d5] /usr/sbin/mysqld(lob::first_page_t::load_x(page_id_t const&, page_size_t const&, mtr_t*)+0x34) [0x55d2b6664ec4] /usr/sbin/mysqld(lob::purge(lob::DeleteContext*, dict_index_t*, unsigned long, unsigned long, unsigned long, upd_field_t const*, purge_node_t*)+0x5ac) [0x55d2b666a1bc] /usr/sbin/mysqld(lob::BtrContext::free_externally_stored_fields(unsigned long, unsigned long, bool, unsigned long, purge_node_t*)+0x284) [0x55d2b62edfd4] /usr/sbin/mysqld(btr_cur_pessimistic_delete(dberr_t*, bool, btr_cur_t*, unsigned int, bool, unsigned long, unsigned long, unsigned long, mtr_t*, btr_pcur_t*, purge_node_t*)+0x2fd) [0x55d2b64ca8dd] /usr/sbin/mysqld(+0x266f635) [0x55d2b6681635] /usr/sbin/mysqld(row_undo_ins(undo_node_t*, que_thr_t*)+0x7c3) [0x55d2b66830a3] /usr/sbin/mysqld(row_undo_step(que_thr_t*)+0x49f) [0x55d2b63faf0f] /usr/sbin/mysqld(que_run_threads(que_thr_t*)+0xc88) [0x55d2b639b138] /usr/sbin/mysqld(+0x244a737) [0x55d2b645c737] /usr/sbin/mysqld(trx_rollback_to_savepoint(trx_t*, trx_savept_t*)+0x2a) [0x55d2b645c9ea] /usr/sbin/mysqld(trx_rollback_to_savepoint_for_mysql(trx_t*, char const*, long*)+0x8b) [0x55d2b645cb4b] /usr/sbin/mysqld(+0x228db82) [0x55d2b629fb82] /usr/sbin/mysqld(ha_rollback_to_savepoint(THD*, SAVEPOINT*)+0xb2) [0x55d2b4fccd42] /usr/sbin/mysqld(trans_rollback_to_savepoint(THD*, MYSQL_LEX_STRING)+0x53) [0x55d2b54024d3] /usr/sbin/mysqld(mysql_execute_command(THD*, bool)+0x4cae) [0x55d2b52d40be] /usr/sbin/mysqld(dispatch_sql_command(THD*, Parser_state*, bool)+0x4c3) [0x55d2b52d7383] /usr/sbin/mysqld(+0x12c594c) [0x55d2b52d794c] /usr/sbin/mysqld(dispatch_command(THD*, COM_DATA const*, enum_server_command)+0x3ca6) [0x55d2b52dc476] /usr/sbin/mysqld(do_command(THD*)+0x1b4) [0x55d2b52dca44] /usr/sbin/mysqld(+0x141b488) [0x55d2b542d488] /usr/sbin/mysqld(+0x26bc1d2) [0x55d2b66ce1d2] /lib/x86_64-linux-gnu/libpthread.so.0(+0x7ea7) [0x7f116c77eea7] /lib/x86_64-linux-gnu/libc.so.6(clone+0x3f) [0x7f116bf2baef] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (7f11085ffe40): ROLLBACK TO SAVEPOINT active_record_1 Connection ID (thread ID): 811016 Status: NOT_KILLED You may download the Percona XtraDB Cluster operations manual by visiting http://www.percona.com/software/percona-xtradb-cluster/. You may find information in the manual which will help you identify the cause of the crash. 2022-11-16T15:21:25.902075Z 811016 [Note] [MY-000000] [WSREP] Initiating SST cancellation Before becoming aware of this bug report I had opened https://jira.percona.com/browse/PS-8489 The code location seems to be equivalent, inside the buf_block_init_low function: * https://github.com/mysql/mysql-server/blob/mysql-cluster-8.0.30/storage/innobase/buf/buf0b... * https://github.com/percona/percona-xtradb-cluster/blob/8.0/storage/innobase/buf/buf0buf.cc...
[18 Nov 2022 4:45]
Johannes Pretorius
I can confirm that this was a table corruption but not caused by Hardware as the same corruption actually replicated out to the slave for the same table. Thus it is inside mysql due to the replication having the exact same error and issue (replication server is on a server on AWS, diff hardware etc). I had to move the table that created the issue out of the database and re-create could not even with all recovery methods get anything done with it . Doing a normal select statement from the table even crashed the server Note I can mention that this was the only table on the whole database that partitioned the table. I reverted back to standard table after the crash and no problems anymore It had 6 Partitions, based on LINEAR_KEY on the primary key
[18 Nov 2022 4:46]
Johannes Pretorius
Corrupt table Partitioning that crashed backup
Attachment: partition_corruption.jpg (image/jpeg, text), 67.63 KiB.
[18 Nov 2022 14:07]
MySQL Verification Team
Hi Mr. Pretorius and Wachowski, Hi Mr. Pretorius, Thank you for your bug report. What we need is a repeatable test case in the form of the set of SQL statements that always lead to this corruption. If you can not provide a test case of that type, then we can not process this bug report, at all. However, if you can provide us with a repeatable test case, then we would denote it as a Security Vulnerability, which would then put this report on the faster track. Please read further on regarding how to prepare a describe report for us ...... Not enough information was provided for us to be able to handle this bug. Please re-read the instructions at http://bugs.mysql.com/how-to-report.php If you can provide more information, feel free to add it to this bug and we will change the status back to 'Open'. Thank you for your interest in MySQL.
[3 Feb 2023 12:51]
null null
2023-01-31T15:55:29.065338+08:00 6835913 [Warning] [MY-013021] [InnoDB] A transaction id in a record of table `ftmarket_mp_test`.`tplse_tm_activity_info` is newer than the system-wide maximum. 2023-01-31T15:55:29.082298+08:00 6835913 [Warning] [MY-013021] [InnoDB] A transaction id in a record of table `ftmarket_mp_test`.`tplse_tm_activity_info` is newer than the system-wide maximum. 2023-01-31T15:55:29.083822+08:00 6835913 [ERROR] [MY-013183] [InnoDB] Assertion failure: buf0buf.cc:3224:block->page.get_space() != nullptr thread 47559250720512 InnoDB: We intentionally generate a memory trap. InnoDB: Submit a detailed bug report to http://bugs.mysql.com. InnoDB: If you get repeated assertion failures or crashes, even InnoDB: immediately after the mysqld startup, there may be InnoDB: corruption in the InnoDB tablespace. Please refer to InnoDB: http://dev.mysql.com/doc/refman/8.0/en/forcing-innodb-recovery.html InnoDB: about forcing recovery. 07:55:29 UTC - mysqld got signal 6 ; Most likely, you have hit a bug, but this error can also be caused by malfunctioning hardware. Thread pointer: 0x2b4024038e40 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 2b41403ead40 thread_stack 0x80000 /usr/local/mysql/bin/mysqld(my_print_stacktrace(unsigned char const*, unsigned long)+0x2e) [0x1fbe99e] /usr/local/mysql/bin/mysqld(print_fatal_signal(int)+0x2eb) [0x106677b] /usr/local/mysql/bin/mysqld(my_server_abort()+0x5e) [0x106687e] /usr/local/mysql/bin/mysqld(my_abort()+0xa) [0x1fb8d3a] /usr/local/mysql/bin/mysqld(ut_dbg_assertion_failed(char const*, char const*, unsigned long)+0x30c) [0x229515c] /usr/local/mysql/bin/mysqld() [0x22f504b] /usr/local/mysql/bin/mysqld(buf_page_init_for_read(dberr_t*, unsigned long, page_id_t const&, page_size_t const&, bool)+0x29f) [0x22fd08f] /usr/local/mysql/bin/mysqld(buf_read_page_low(dberr_t*, bool, unsigned long, unsigned long, page_id_t const&, page_size_t const&, bool)+0x86) [0x232dac6] /usr/local/mysql/bin/mysqld(buf_read_page(page_id_t const&, page_size_t const&)+0x37) [0x232e177] /usr/local/mysql/bin/mysqld(Buf_fetch<Buf_fetch_normal>::read_page()+0x26) [0x22f50c6] /usr/local/mysql/bin/mysqld(Buf_fetch_normal::get(buf_block_t*&)+0x47c) [0x22fdcdc] /usr/local/mysql/bin/mysqld(Buf_fetch<Buf_fetch_normal>::single_page()+0x46) [0x22fdd66] /usr/local/mysql/bin/mysqld(buf_page_get_gen(page_id_t const&, page_size_t const&, unsigned long, buf_block_t*, Page_fetch, ut::Location, mtr_t*, bool)+0x1b3) [0x22ff013] /usr/local/mysql/bin/mysqld() [0x2267104] /usr/local/mysql/bin/mysqld(trx_undo_prev_version_build(unsigned char const*, mtr_t*, unsigned char const*, dict_index_t const*, unsigned long*, mem_block_info_t*, unsigned char**, mem_block_info_t*, dtuple_t const**, unsigned long, lob::undo_vers_t*)+0x25e) [0x226756e] /usr/local/mysql/bin/mysqld(row_vers_build_for_consistent_read(unsigned char const*, mtr_t*, dict_index_t*, unsigned long**, ReadView*, mem_block_info_t**, mem_block_info_t*, unsigned char**, dtuple_t const**, lob::undo_vers_t*)+0x22a) [0x222469a] /usr/local/mysql/bin/mysqld(row_search_mvcc(unsigned char*, page_cur_mode_t, row_prebuilt_t*, unsigned long, unsigned long)+0x2e5e) [0x221012e] /usr/local/mysql/bin/mysqld(ha_innobase::general_fetch(unsigned char*, unsigned int, unsigned int)+0x1e0) [0x20a7cf0] /usr/local/mysql/bin/mysqld(handler::ha_rnd_next(unsigned char*)+0x147) [0x116a477] /usr/local/mysql/bin/mysqld(TableScanIterator::Read()+0x1d) [0x12b138d] /usr/local/mysql/bin/mysqld(FilterIterator::Read()+0x14) [0x1403794] /usr/local/mysql/bin/mysqld(NestedLoopIterator::Read()+0xa7) [0x1403947] /usr/local/mysql/bin/mysqld(NestedLoopIterator::Read()+0xa7) [0x1403947] /usr/local/mysql/bin/mysqld(MaterializeIterator<DummyIteratorProfiler>::MaterializeQueryBlock(materialize_iterator::QueryBlock const&, unsigned long long*)+0x197) [0x1408207] /usr/local/mysql/bin/mysqld(MaterializeIterator<DummyIteratorProfiler>::Init()+0x325) [0x1408b45] /usr/local/mysql/bin/mysqld(AggregateIterator::Init()+0x27) [0x14056d7] /usr/local/mysql/bin/mysqld(Query_expression::ExecuteIteratorQuery(THD*)+0x345) [0xfd3a75] /usr/local/mysql/bin/mysqld(Query_expression::execute(THD*)+0x2c) [0xfd3cdc] /usr/local/mysql/bin/mysqld(Sql_cmd_dml::execute(THD*)+0x2c5) [0xf60fb5] /usr/local/mysql/bin/mysqld(mysql_execute_command(THD*, bool)+0xad8) [0xeff0f8] /usr/local/mysql/bin/mysqld(dispatch_sql_command(THD*, Parser_state*)+0x403) [0xf04163] /usr/local/mysql/bin/mysqld(dispatch_command(THD*, COM_DATA const*, enum_server_command)+0x2080) [0xf066f0] /usr/local/mysql/bin/mysqld(do_command(THD*)+0x1df) [0xf076af] /usr/local/mysql/bin/mysqld() [0x10574b8] /usr/local/mysql/bin/mysqld() [0x2526cda] /lib64/libpthread.so.0(+0x7ea5) [0x2b337410cea5] /lib64/libc.so.6(clone+0x6d) [0x2b3375e3f96d] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (2b40247abbb0): SELECT COUNT(1) FROM ( SELECT ttai.entity_id activityId, ttai.activity_title, ttai.organ_id, if(ttai.organ_id != '1' AND locate(ttai.organ_id, '1') > 0, 'Y', 'N') upOrganFlag, ttai.activity_code, ttai.parent_id, ttai.activity_start_date, ttai.activity_end_date, ttai.activity_type, ttas.unshelve_flag, ttas.entity_id sessionId, ttas.session_code, ttas.session_name, ttas.session_state, ttas.session_start_date, ttas.session_end_date, (SELECT count(DISTINCT taof.open_id) AS cnt FROM v_tplse_activity_operation_flow taof WHERE taof.operation_type = 'visit' AND taof.activity_code IS NOT NULL AND taof.activity_code = ttai.activity_code) visitCustCnt, tt.appointmentCustCnt, tt.signCustCnt, ttas.create_time, ttas.create_uname, ttas.address_info FROM tplse_tm_activity_info ttai INNER JOIN tplse_tm_activity_session ttas ON ttai.entity_id = ttas.activity_id LEFT JOIN (SELECT t.batch_no, sum(if(t.operation_type = 'order', cnt, 0)) appointmentCustCnt, sum(if(t.operation_type = 'sign', cnt, 0)) signCustCnt FROM (SELECT taof.b Connection ID (thread ID): 6835913 Status: NOT_KILLED The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash.
[6 Feb 2023 14:11]
MySQL Verification Team
Hi, What you have sent above is not a repeatable test case.
