MySQL Bugs: #115240: failed to apply an redo on page in file mysql.ibd and caused crashes

Bug #115240	failed to apply an redo on page in file mysql.ibd and caused crashes
Submitted:	6 Jun 2024 7:59	Modified:	19 Jun 2024 11:11
Reporter:	gang liu	Email Updates:
Status:	Verified	Impact on me:	None
Category:	MySQL Server: Clone Plugin	Severity:	S6 (Debug Builds)
Version:	8.0	OS:	Any
Assigned to:		CPU Architecture:	Any
Tags:	clone plugin, page-tracking

Description:
after 'clone instance...' success, recover the mysql instance using the clone data will cause crashes because it cannot apply redo on one page in file mysql.ibd. 

How to repeat:
1. add receipt_clone.test in directory `mysql-test/suite/clone/t/`;

receipt_clone.test:
```
--source include/have_debug.inc
--source include/have_innodb_default_undo_tablespaces.inc
--source ../include/clone_connection_begin.inc

--let $CLONE_DATADIR = $MYSQL_TMP_DIR/data_new
--let $CLONE_DATADIR1 = $MYSQL_TMP_DIR/data_new1

--source include/count_sessions.inc

## Install plugin

--replace_result $CLONE_PLUGIN CLONE_PLUGIN
--eval INSTALL PLUGIN clone SONAME '$CLONE_PLUGIN'

INSTALL COMPONENT "file://component_mysqlbackup";
SELECT mysqlbackup_page_track_set(true);

## Create test schema
CREATE TABLE t1(col1 INT PRIMARY KEY, col2 int, col3 varchar(64), col4 BLOB);

CREATE TABLE t2(col1 INT PRIMARY KEY, col2 int, col3 varchar(64), col4 BLOB)
PARTITION BY KEY(col1) PARTITIONS 5;

insert into t1 values(1,1,1,"aaa");
insert into t1 values(2,2,2,"aaa");
insert into t1 values(3,3,3,"aaa");
insert into t1 values(4,4,4,"aaa");
insert into t1 values(5,5,5,"aaa");
insert into t1 values(6,6,6,"aaa");

insert into t2 values(1,1,1,"aaa");
insert into t2 values(2,2,2,"aaa");
insert into t2 values(3,3,3,"aaa");
insert into t2 values(4,4,4,"aaa");

CREATE INDEX index_col2 ON t1 (col2);
CREATE INDEX index_col2 ON t2 (col2);

# Check base rows
SHOW CREATE TABLE t1;
SELECT count(*) from t1;
SELECT col1, col2, col3 FROM t1 ORDER BY col1;
SELECT col1, col2, col3 FROM t1 ORDER BY col1;

SHOW CREATE TABLE t2;
SELECT count(*) from t2;
SELECT col1, col2, col3 FROM t2 ORDER BY col1;
SELECT col1, col2, col3 FROM t2 ORDER BY col1 DESC;

--echo # In connection con1 - Running Insert
connect (con1,localhost,root,,);
begin;
--send insert into t1 values(120,120,120,"aaa")

--echo # In connection default - Cloning database
--connection clone_conn_1

SELECT count(*) from t1;
SELECT col1, col2, col3 FROM t1 ORDER BY col1;
SELECT col1, col2, col3 FROM t1 ORDER BY col1;

SHOW CREATE TABLE t2;
SELECT count(*) from t2;
SELECT col1, col2, col3 FROM t2 ORDER BY col1;
SELECT col1, col2, col3 FROM t2 ORDER BY col1;

--echo # In connection con1 - Waiting
connection con1;
--reap
commit;

--echo # In connection default
connection default;
disconnect con1;

select ID, STATE, ERROR_NO from performance_schema.clone_status;
select ID, STAGE, STATE  from performance_schema.clone_progress;

--connection default
select * from information_schema.INNODB_TABLESPACES;
drop table t1;
drop table t2;

# Clean recipient by cloning the cleaned up donor

--source ../include/clone_connection_end.inc

UNINSTALL COMPONENT "file://component_mysqlbackup";
UNINSTALL PLUGIN clone;

```

2. run `./mtr  clone.monitor_progress clone.receipt_courrency  clone.remote_compress clone.remote_ddl_alter_copy --record`

if apply the diff:
```
diff --git a/storage/innobase/buf/buf0flu.cc b/storage/innobase/buf/buf0flu.cc
index cc510f3c461..f9ffd5abea2 100644
--- a/storage/innobase/buf/buf0flu.cc
+++ b/storage/innobase/buf/buf0flu.cc
@@ -1353,7 +1353,7 @@ bool buf_flush_page(buf_pool_t *buf_pool, buf_page_t *bpage,
       lsn_t frame_lsn = mach_read_from_8(frame + FIL_PAGE_LSN);
 
       arch_page_sys->track_page(bpage, buf_pool->track_page_lsn, frame_lsn,
-                                false);
+                                true);
     }

```
 `./mtr  clone.monitor_progress clone.receipt_courrency  clone.remote_compress clone.remote_ddl_alter_copy --record` will be sucess.

the diff will make page-tracking system track the same pageID many time.

Backtrace：

```
024-06-06T07:07:29.612201Z 1 [Note] [MY-012533] [InnoDB] 60%
2024-06-06T07:07:29.672459Z 0 [ERROR] [MY-013183] [InnoDB] Assertion failure: page0page.ic:533:page_offset(rec) <= page_header_get_field(page, PAGE_HEAP_TOP) thread 140266961839872
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: http://dev.mysql.com/doc/refman/8.0/en/forcing-innodb-recovery.html
InnoDB: about forcing recovery.
2024-06-06T07:07:29Z UTC - mysqld got signal 6 ;
Most likely, you have hit a bug, but this error can also be caused by malfunctioning hardware.
BuildID[sha1]=94e2e89670d7fbd1f9f9ed9567fcddb329b4f729
Thread pointer: 0x0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0 thread_stack 0x100000
2024-06-06T07:07:29.673586Z 0 [ERROR] [MY-013183] [InnoDB] Assertion failure: page0page.ic:533:page_offset(rec) <= page_header_get_field(page, PAGE_HEAP_TOP) thread 140266951350016
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: http://dev.mysql.com/doc/refman/8.0/en/forcing-innodb-recovery.html
InnoDB: about forcing recovery.
2024-06-06T07:07:29.675369Z 0 [ERROR] [MY-013183] [InnoDB] Assertion failure: page0cur.cc:1330:rec_get_status(current_rec) <= REC_STATUS_INFIMUM thread 140266940860160
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: http://dev.mysql.com/doc/refman/8.0/en/forcing-innodb-recovery.html
InnoDB: about forcing recovery.
2024-06-06T07:07:29.678292Z 0 [ERROR] [MY-013183] [InnoDB] Assertion failure: page0cur.cc:1323:current_rec != insert_rec thread 140266652563200
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: http://dev.mysql.com/doc/refman/8.0/en/forcing-innodb-recovery.html
InnoDB: about forcing recovery.
2024-06-06T07:07:29.682672Z 1 [Note] [MY-012533] [InnoDB] 70%
/home/liugang/code/percona-server/bld_debug_mysql8032/install_debug/bin/mysqld(my_print_stacktrace(unsigned char const*, unsigned long)+0x59) [0x55d894a90d52]
/home/liugang/code/percona-server/bld_debug_mysql8032/install_debug/bin/mysqld(print_fatal_signal(int)+0x38e) [0x55d8934d2c43]
/home/liugang/code/percona-server/bld_debug_mysql8032/install_debug/bin/mysqld(my_server_abort()+0x78) [0x55d8934d2ef9]
/home/liugang/code/percona-server/bld_debug_mysql8032/install_debug/bin/mysqld(my_abort()+0x11) [0x55d894a86c33]
/home/liugang/code/percona-server/bld_debug_mysql8032/install_debug/bin/mysqld(ut_dbg_assertion_failed(char const*, char const*, unsigned long)+0x1e7) [0x55d894f9891a]
/home/liugang/code/percona-server/bld_debug_mysql8032/install_debug/bin/mysqld(+0x531bbbb) [0x55d894ddabbb]
/home/liugang/code/percona-server/bld_debug_mysql8032/install_debug/bin/mysqld(+0x531b788) [0x55d894dda788]
/home/liugang/code/percona-server/bld_debug_mysql8032/install_debug/bin/mysqld(page_cur_delete_rec(page_cur_t*, dict_index_t const*, unsigned long const*, mtr_t*)+0x1ad) [0x55d894de29cd]
/home/liugang/code/percona-server/bld_debug_mysql8032/install_debug/bin/mysqld(page_cur_parse_delete_rec(unsigned char*, unsigned char*, buf_block_t*, dict_index_t*, mtr_t*)+0x233) [0x55d894de27e0]
/home/liugang/code/percona-server/bld_debug_mysql8032/install_debug/bin/mysqld(+0x52bdd61) [0x55d894d7cd61]
/home/liugang/code/percona-server/bld_debug_mysql8032/install_debug/bin/mysqld(recv_recover_page_func(bool, buf_block_t*)+0x8a7) [0x55d894d7e486]
/home/liugang/code/percona-server/bld_debug_mysql8032/install_debug/bin/mysqld(+0x5581f63) [0x55d895040f63]
/home/liugang/code/percona-server/bld_debug_mysql8032/install_debug/bin/mysqld(buf_page_io_complete(buf_page_t*, bool)+0x814) [0x55d8950536d9]
/home/liugang/code/percona-server/bld_debug_mysql8032/install_debug/bin/mysqld(fil_aio_wait(unsigned long)+0x23d) [0x55d8951eadbc]
/home/liugang/code/percona-server/bld_debug_mysql8032/install_debug/bin/mysqld(+0x5436855) [0x55d894ef5855]
/home/liugang/code/percona-server/bld_debug_mysql8032/install_debug/bin/mysqld(void std::__invoke_impl<void, void (*&)(unsigned long), unsigned long&>(std::__invoke_other, void (*&)(unsigned long), unsigned long&)+0x38) [0x55d894f0d6d5]
/home/liugang/code/percona-server/bld_debug_mysql8032/install_debug/bin/mysqld(std::__invoke_result<void (*&)(unsigned long), unsigned long&>::type std::__invoke<void (*&)(unsigned long), unsigned long&>(void (*&)(unsigned long), unsigned long&)+0x3b) [0x55d894f0d62e]
/home/liugang/code/percona-server/bld_debug_mysql8032/install_debug/bin/mysqld(void std::_Bind<void (*(unsigned long))(unsigned long)>::__call<void, , 0ul>(std::tuple<>&&, std::_Index_tuple<0ul>)+0x5b) [0x55d894f0d575]
/home/liugang/code/percona-server/bld_debug_mysql8032/install_debug/bin/mysqld(void std::_Bind<void (*(unsigned long))(unsigned long)>::operator()<, void>()+0x37) [0x55d894f0d49b]
/home/liugang/code/percona-server/bld_debug_mysql8032/install_debug/bin/mysqld(void Detached_thread::operator()<void (*)(unsigned long), unsigned long>(void (*&&)(unsigned long), unsigned long&&)+0xdf) [0x55d894f0d391]
/home/liugang/code/percona-server/bld_debug_mysql8032/install_debug/bin/mysqld(void std::__invoke_impl<void, Detached_thread, void (*)(unsigned long), unsigned long>(std::__invoke_other, Detached_thread&&, void (*&&)(unsigned long), unsigned long&&)+0x53) [0x55d894f0d26b]
/home/liugang/code/percona-server/bld_debug_mysql8032/install_debug/bin/mysqld(std::__invoke_result<Detached_thread, void (*)(unsigned long), unsigned long>::type std::__invoke<Detached_thread, void (*)(unsigned long), unsigned long>(Detached_thread&&, void (*&&)(unsigned long), unsigned long&&)+0x53) [0x55d894f0d190]
/home/liugang/code/percona-server/bld_debug_mysql8032/install_debug/bin/mysqld(void std::thread::_Invoker<std::tuple<Detached_thread, void (*)(unsigned long), unsigned long> >::_M_invoke<0ul, 1ul, 2ul>(std::_Index_tuple<0ul, 1ul, 2ul>)+0x63) [0x55d894f0d0af]
/home/liugang/code/percona-server/bld_debug_mysql8032/install_debug/bin/mysqld(std::thread::_Invoker<std::tuple<Detached_thread, void (*)(unsigned long), unsigned long> >::operator()()+0x1c) [0x55d894f0d048]
/home/liugang/code/percona-server/bld_debug_mysql8032/install_debug/bin/mysqld(std::thread::_State_impl<std::thread::_Invoker<std::tuple<Detached_thread, void (*)(unsigned long), unsigned long> > >::_M_run()+0x20) [0x55d894f0d028]
/lib/x86_64-linux-gnu/libstdc++.so.6(+0xe6793) [0x7f9281f1e793]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x8609) [0x7f92820b4609]
/lib/x86_64-linux-gnu/libc.so.6(clone+0x43) [0x7f9281bef353]

```

Hi Mr. liu,

Thank you very much for your bug report.

However, we have to ask you to provide us with the additional info.

First of all, this is a forum for MySQL products, which includes its server and its plugins. Hence, please provide your test case for MySQL, which means that you should repeat the behaviour on the official MySQL binaries, available from https://dev.mysql.com.

Next and much more important. Version 8.2 is already deprecated. Hence, we need a test case for the current version, which is 8.4.

Needless to say, we are puzzled regarding what you are using. Version 8.2 had only a release 8.2.0. We never had a release 8.2.32 ....

We truly do not know what have you tested your problem with.

Meanwhile, we shall try to repeat the behaviour with MySQL 8.4.

We are waiting on your feedback.

Hi Mr. liu,

We can not proceed with repeating your test case, since our 8.0.37 and our 8.4.0 distributions do not have the include or test named either 'clone.receipt_courrency' nor  'clone.receipt_currency'.

Can't repeat.

1.I am sorry，in this I provide error informaiton：
 receipt_clone.test ===> receipt_currency.test

2.`receipt_currency.test` is that I add test script. however, it should not cause core.

Hi Mr. liu,

We are not able to repeat your test case.

This is what we get:

==============================================================================
                  TEST NAME                       RESULT  TIME (ms) COMMENT
------------------------------------------------------------------------------
[ 20%] clone.monitor_progress                    [ skipped ]  Test needs debug binaries.
[ 40%] clone.receipt_currency                    [ skipped ]  Test needs debug binaries.
[ 60%] clone.remote_compress                     [ skipped ]  Test requires 'Punch hole support'
[ 80%] clone.remote_ddl_alter_copy               [ skipped ]  Test requires 'have_debug_sync'
[100%] shutdown_report                           [ pass ]
------------------------------------------------------------------------------
The servers were restarted 0 times
The servers were reinitialized 0 times
Spent 0.000 of 11 seconds executing testcases

Completed: All 1 tests were successful.

my mistake. I repeat the bug in mysql-8.0.32 debug version.

my mistake. I repeat the bug in mysql-8.0.32 debug version.

Hi Mr. liu,

We tried with 8.0.37 debug release and could not repeat your test case, either ......

==============================================================================
                 TEST NAME                       RESULT  TIME (ms)
COMMENT
------------------------------------------------------------------------------
[ 20%] clone.monitor_progress                           [ pass ]
[ 40%] clone.receipt_currency                           [ pass ]
[ 60%] clone.remote_compress                           [ pass ]
[ 80%] clone.remote_ddl_alter_copy                           [ pass ]
[100%] shutdown_report                           [ pass ]
------------------------------------------------------------------------------

8.0.32 is much older release then 8.0.37 of the server version 8.0.

Hence, we only test latest stable releases.

Can't repeat.

I use mysql-8.0.37 debug version, repeat the bug stably.

I use the following camke command:

``` 
cmake ../ -DCMAKE_BUILD_TYPE=debug -DCMAKE_INSTALL_PREFIX=`pwd`/install_debug -DFEATURE_SET=community -DWITH_PAM=ON -DWITH_KEYRING_VAULT=OFF -DWITH_INNODB_MEMCACHED=ON -DWITH_ZLIB=bundled -DWITH_NUMA=ON -DENABLE_DOWNLOADS=1 -DWITH_UNIT_TESTS=OFF -DWITH_EMBEDDED_SERVER=OFF -DWITH_LIdBEVENT=bundled -DWITH_SSL=system -Dsomedefine++=1 -DDOWNLOAD_BOOST=1 -DWITH_BOOST=`pwd`/../boost
```

liugang@liugang-MS-7C82:~/code/mysql-server/bld_debug/install_debug/mysql-test$ ./mtr  clone.monitor_progress clone.receipt_courrency  clone.remote_compress clone.remote_ddl_alter_copy --record
Logging: ./mtr  clone.monitor_progress clone.receipt_courrency clone.remote_compress clone.remote_ddl_alter_copy --record
MySQL Version 8.0.37
Checking supported features
 - Binaries are debug compiled
Using 'all' suites
Collecting tests
Removing old var directory
Creating var directory '/home/liugang/code/mysql-server/bld_debug/install_debug/mysql-test/var'
Installing system database
Using parallel: 1

==============================================================================
                  TEST NAME                       RESULT  TIME (ms) COMMENT
------------------------------------------------------------------------------
[ 20%] clone.monitor_progress                    [ pass ]   7343
[ 40%] clone.receipt_courrency                   [ pass ]    671
[ 60%] clone.remote_compress                     [ pass ]  63786
[ 80%] clone.remote_ddl_alter_copy               [ fail ]
        Test ended at 2024-06-12 11:17:28

the core:
24-06-12T03:17:27.775597Z 1 [Note] [MY-012533] [InnoDB] 50%
2024-06-12T03:17:27.809119Z 1 [Note] [MY-012533] [InnoDB] 60%
2024-06-12T03:17:27.836176Z 0 [ERROR] [MY-013183] [InnoDB] Assertion failure: page0page.ic:534:page_offset(rec) <= page_header_get_field(page, PAGE_HEAP_TOP) thread 139747789969152
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: http://dev.mysql.com/doc/refman/8.0/en/forcing-innodb-recovery.html
InnoDB: about forcing recovery.
2024-06-12T03:17:27Z UTC - mysqld got signal 6 ;
Most likely, you have hit a bug, but this error can also be caused by malfunctioning hardware.
BuildID[sha1]=1077c5527cfa773e743194ee665e7c3f3703b9e2
Thread pointer: 0x0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0 thread_stack 0x100000
2024-06-12T03:17:27.840513Z 0 [ERROR] [MY-013183] [InnoDB] Assertion failure: page0cur.cc:1207 thread 139747779479296
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: http://dev.mysql.com/doc/refman/8.0/en/forcing-innodb-recovery.html
InnoDB: about forcing recovery.
2024-06-12T03:17:27.848915Z 0 [ERROR] [MY-013183] [InnoDB] Assertion failure: page0page.cc:111:r >= page + (PAGE_DATA + 2 * REC_N_NEW_EXTRA_BYTES + 8) thread 139747565500160
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: http://dev.mysql.com/doc/refman/8.0/en/forcing-innodb-recovery.html
InnoDB: about forcing recovery.
2024-06-12T03:17:27.850138Z 0 [ERROR] [MY-013183] [InnoDB] Assertion failure: page0page.cc:111:r >= page + (PAGE_DATA + 2 * REC_N_NEW_EXTRA_BYTES + 8) thread 139747555010304
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: http://dev.mysql.com/doc/refman/8.0/en/forcing-innodb-recovery.html
InnoDB: about forcing recovery.
2024-06-12T03:17:27.854261Z 1 [Note] [MY-012533] [InnoDB] 70%
/home/liugang/code/mysql-server/bld_debug/install_debug/bin/mysqld(my_print_stacktrace(unsigned char const*, unsigned long)+0x59) [0x55c76ba29fd5]
/home/liugang/code/mysql-server/bld_debug/install_debug/bin/mysqld(print_fatal_signal(int)+0x38e) [0x55c76a45a8d7]
/home/liugang/code/mysql-server/bld_debug/install_debug/bin/mysqld(my_server_abort()+0x78) [0x55c76a45ab8d]
/home/liugang/code/mysql-server/bld_debug/install_debug/bin/mysqld(my_abort()+0x11) [0x55c76ba1fe63]
/home/liugang/code/mysql-server/bld_debug/install_debug/bin/mysqld(ut_dbg_assertion_failed(char const*, char const*, unsigned long)+0x1e7) [0x55c76be5840c]
/home/liugang/code/mysql-server/bld_debug/install_debug/bin/mysqld(+0x52cb1b3) [0x55c76bc981b3]
/home/liugang/code/mysql-server/bld_debug/install_debug/bin/mysqld(+0x52caddc) [0x55c76bc97ddc]
/home/liugang/code/mysql-server/bld_debug/install_debug/bin/mysqld(page_cur_insert_rec_low(unsigned char*, dict_index_t*, unsigned char const*, unsigned long*, mtr_t*)+0x1dc) [0x55c76bc9c979]
/home/liugang/code/mysql-server/bld_debug/install_debug/bin/mysqld(+0x52cc2eb) [0x55c76bc992eb]
/home/liugang/code/mysql-server/bld_debug/install_debug/bin/mysqld(page_cur_parse_insert_rec(bool, unsigned char const*, unsigned char const*, buf_block_t*, dict_index_t*, mtr_t*)+0x907) [0x55c76bc9c6e3]
/home/liugang/code/mysql-server/bld_debug/install_debug/bin/mysqld(+0x526c553) [0x55c76bc39553]
/home/liugang/code/mysql-server/bld_debug/install_debug/bin/mysqld(recv_recover_page_func(bool, buf_block_t*)+0x8a7) [0x55c76bc3c2f8]
/home/liugang/code/mysql-server/bld_debug/install_debug/bin/mysqld(+0x5533e73) [0x55c76bf00e73]
/home/liugang/code/mysql-server/bld_debug/install_debug/bin/mysqld(buf_page_io_complete(buf_page_t*, bool)+0x814) [0x55c76bf135d5]
/home/liugang/code/mysql-server/bld_debug/install_debug/bin/mysqld(fil_aio_wait(unsigned long)+0x23d) [0x55c76c0ae4a8]
/home/liugang/code/mysql-server/bld_debug/install_debug/bin/mysqld(+0x52b2f43) [0x55c76bc7ff43]
/home/liugang/code/my

Hi Mr. liu,

We can't repeat your report.

First of all, it is our policy that we do not support test cases originating from our clones. Failure to those test cases should be reported to the maintainer of that release.

Next, we do not have support for the punch hole. Hence, one more reason to report your problems to the correct place.

I analyze the bug:

***How to repeat:***
1.I start clone with the following command:

>clone local data directory='/home/liugang/code/data/clone1'

before the command finish, mysql server will remove the file `#ib_archive/page_group_20428265` , but not remove the file `#ib_archive/ib_dblwr/dblwr_0` or overwrite with zeros.

2.I start page tracking with the following function:

>SELECT mysqlbackup_page_track_set(true);

then, I do some DML opperation in the mysql server.

3. after the system lsn = checkpoing lsn, shutdown the mysqld:

> SHUTDOWN;

4. restart the mysqld again.

the mysqld will use file `#ib_archive/ib_dblwr/dblwr_0` in step1 to recover the file `#ib_archive/page_group_20469849/ib_page_0` during recover.

***Analyze:***

the file #ib_archive/ib_dblwr/dblwr_0 was divided into three parts, each part size is 16K：

```c++
/** Page Archive doublewrite buffer block offsets */
enum Arch_Page_Dblwr_Offset {
/** Archive doublewrite buffer page offset for RESET page. */
ARCH_PAGE_DBLWR_RESET_PAGE = 0,
/* Archive doublewrite buffer page offset for FULL FLUSH page. */
ARCH_PAGE_DBLWR_FULL_FLUSH_PAGE,
/* Archive doublewrite buffer page offset for PARTIAL FLUSH page. */
ARCH_PAGE_DBLWR_PARTIAL_FLUSH_PAGE
};
```

Before the clone finish, page tracking will use ARCH_PAGE_DBLWR_FULL_FLUSH_PAGE to flush data to #ib_archive/page_group_20428265/ib_page_0, even though the tracking data size is less then 16K. with clone finished, #ib_archive/page_group_20428265/ib_page_0 was deleted, the file `#ib_archive/ib_dblwr/dblwr_0 ` still keep the data.

After I finish step2, the tracking data is less then 16K in memory , the file #ib_archive/ib_dblwr/dblwr_0 will not use rewite the ARCH_PAGE_DBLWR_FULL_FLUSH_PAGE part event though I shutdown mysqld.

When I start the mysqld, It will use ARCH_PAGE_DBLWR_FULL_FLUSH_PAGE part (which is tracking data in clone) to recover the #ib_archive/page_group_20428265/ib_page_0 .

Hi Mr. liu,

Thank you for your last test case.

However, the original test case that you sent us passed successfully on 8.0.37 production binary.

We can not repeat your latest test case.

We do not have directories that you are using .....

Step 3 is impossible to repeat, since we do not see how we can find out about the lsn 

Next, you are asking us to inspect the memory and file contents.

Hence, this is not truly a test case that we are looking for.

Overall, we require just a test case to run and the error to show up.

The MTR test case that you have sent us worked just fine .......

We just need a repeatable test case and not a whole debugging session.

Can't repeat.

Hi Mr. liu,

We have engaged a number of developers on this bug and they have managed to reproduce the crash on debug binaries only.

Verified with a severity applicable to debug builds only.

This bug affects 8.0 and all higher versions.

Thank you for your contribution.

Thank you for the replay.
Maybe, the bug only repeat on debug version. the basic reason is that page-tracking infrastructure failed to track the modified pages ID, which lead some increment backup tools based it make an incorrect backup data. then, the cover data is bad.

So, I think this bug is critical!