| Bug #98789 | meb to restore backup may lose data | ||
|---|---|---|---|
| Submitted: | 1 Mar 2020 4:31 | Modified: | 10 May 2020 13:57 |
| Reporter: | Bin Hong (OCA) | Email Updates: | |
| Status: | No Feedback | Impact on me: | |
| Category: | MySQL Enterprise Backup | Severity: | S3 (Non-critical) |
| Version: | 8.0.19 | OS: | Red Hat (7.5) |
| Assigned to: | CPU Architecture: | Any | |
[1 Mar 2020 4:31]
Bin Hong
backup meta data
Attachment: backuplog.tar.gz (application/x-gzip, text), 155.57 KiB.
[1 Mar 2020 4:41]
Bin Hong
fix sysbench use oltp_insert sysbench oltp_insert --mysql_storage_engine = innodb --table-size = 100000 --tables = 1 --mysql-db = test --mysql-port = 3306 --mysql-host = 127.0.0.1 --mysql-user = root- -mysql-password = Actionsky！888 --threads = 10 --time = 100 --report-interval = 1 --events = 0 --db-driver = mysql run
[1 Mar 2020 4:44]
Bin Hong
After copy-back ，exec select query mysql restart 2020-03-01T01:02:06.336326Z 0 [System] [MY-011323] [Server] X Plugin ready for connections. Socket: '/var/run/mysqld/mysqlx.sock' bind-address: '::' port: 33060 02:26:55 UTC - mysqld got signal 11 ; Most likely, you have hit a bug, but this error can also be caused by malfunctioning hardware. Thread pointer: 0x7fd5dc000d40 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 7fd6481adc30 thread_stack 0x46000 /usr/sbin/mysqld(my_print_stacktrace(unsigned char const*, unsigned long)+0x3d) [0x1fb737d] /usr/sbin/mysqld(handle_fatal_signal+0x333) [0xfe0e23] /lib64/libpthread.so.0(+0xf6d0) [0x7fd6696b96d0] /usr/sbin/mysqld(rec_get_offsets_func(unsigned char const*, dict_index_t const*, unsigned long*, unsigned long, mem_block_info_t**)+0x1e) [0x227466e] /usr/sbin/mysqld(Parallel_reader::Scan_ctx::copy_row(unsigned char const*, Parallel_reader::Scan_ctx::Iter*) const+0x3c) [0x21c53fc] /usr/sbin/mysqld(Parallel_reader::Scan_ctx::create_persistent_cursor(page_cur_t const&, mtr_t*) const+0x1ec) [0x21c859c] /usr/sbin/mysqld(Parallel_reader::Scan_ctx::create_range(std::vector<std::pair<std::shared_ptr<Parallel_reader::Scan_ctx::Iter>, std::shared_ptr<Parallel_reader::Scan_ctx::Iter> >, ut_allocator<std::pair<std::shared_ptr<Parallel_reader::Scan_ctx::Iter>, std::shared_ptr<Parallel_reader::Scan_ctx::Iter> > > >&, page_cur_t&, mtr_t*) const+0x39) [0x21cd429] /usr/sbin/mysqld(Parallel_reader::Scan_ctx::create_ranges(Parallel_reader::Scan_range const&, unsigned int, unsigned long, unsigned long, std::vector<std::pair<std::shared_ptr<Parallel_reader::Scan_ctx::Iter>, std::shared_ptr<Parallel_reader::Scan_ctx::Iter> >, ut_allocator<std::pair<std::shared_ptr<Parallel_reader::Scan_ctx::Iter>, std::shared_ptr<Parallel_reader::Scan_ctx::Iter> > > >&, mtr_t*)+0x8f9) [0x21ce139] /usr/sbin/mysqld(Parallel_reader::Scan_ctx::partition(Parallel_reader::Scan_range const&, std::vector<std::pair<std::shared_ptr<Parallel_reader::Scan_ctx::Iter>, std::shared_ptr<Parallel_reader::Scan_ctx::Iter> >, ut_allocator<std::pair<std::shared_ptr<Parallel_reader::Scan_ctx::Iter>, std::shared_ptr<Parallel_reader::Scan_ctx::Iter> > > >&, unsigned long)+0x13d) [0x21cea8d] /usr/sbin/mysqld(Parallel_reader::add_scan(trx_t*, Parallel_reader::Config const&, std::function<dberr_t (Parallel_reader::Ctx const*)>&&)+0x282) [0x21cf762] /usr/sbin/mysqld(row_mysql_parallel_select_count_star(trx_t*, std::vector<dict_index_t*, std::allocator<dict_index_t*> >&, unsigned long, unsigned long*)+0x225) [0x21af615] /usr/sbin/mysqld(row_scan_index_for_mysql(row_prebuilt_t*, dict_index_t*, unsigned long, bool, unsigned long*)+0x2e0) [0x21b9c40] /usr/sbin/mysqld(ha_innobase::records(unsigned long long*)+0x1b5) [0x2079005] /usr/sbin/mysqld(get_exact_record_count(QEP_TAB*, unsigned int, int*)+0x7e) [0xe67b7e] /usr/sbin/mysqld(UnqualifiedCountIterator::Read()+0xa2) [0xe6ca12] /usr/sbin/mysqld(SELECT_LEX_UNIT::ExecuteIteratorQuery(THD*)+0x2db) [0xf6962b] /usr/sbin/mysqld(SELECT_LEX_UNIT::execute(THD*)+0x10a) [0xf6b85a] /usr/sbin/mysqld(Sql_cmd_dml::execute_inner(THD*)+0x22b) [0xef76cb] /usr/sbin/mysqld(Sql_cmd_dml::execute(THD*)+0x418) [0xf00dd8] /usr/sbin/mysqld(mysql_execute_command(THD*, bool)+0x19ed) [0xeb0d7d] /usr/sbin/mysqld(mysql_parse(THD*, Parser_state*)+0x350) [0xeb4ac0] /usr/sbin/mysqld(dispatch_command(THD*, COM_DATA const*, enum_server_command)+0x159b) [0xeb658b] /usr/sbin/mysqld(do_command(THD*)+0x19c) [0xeb7b4c] /usr/sbin/mysqld() [0xfd2240] /usr/sbin/mysqld() [0x24be55c] /lib64/libpthread.so.0(+0x7e25) [0x7fd6696b1e25] /lib64/libc.so.6(clone+0x6d) [0x7fd66783abad] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (7fd5dc09acb8): select count(*) from sbtest1 Connection ID (thread ID): 8 Status: NOT_KILLED The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. 2020-03-01T02:26:57.562977Z 0 [System] [MY-010116] [Server] /usr/sbin/mysqld (mysqld 8.0.19) starting as process 20814
[1 Mar 2020 14:35]
Bin Hong
There may be some problems during the test process, which need to be re-verified, please ignore
[10 Apr 2020 13:56]
MySQL Verification Team
Using the --force option is never recommended. Can you can same problems without using --force?
[11 May 2020 1:00]
Bugs System
No feedback was provided for this bug for over a month, so it is being suspended automatically. If you are able to provide the information that was originally requested, please do so and change the status of the bug back to "Open".
Description: I want to simulate the redo log overwritten scene. Do not enable the redo log archiving, when backing up, run sysbench, and use cgroup to limit the write speed of mysqlbackup program, resulting in redo log overwritten. No errors occurred during the backup process and were successfully recovered, but some of the data was lost. How to repeat: repeat step 1.prepare cgroup and log trigger mkdir -p /sys/fs/cgroup/blkio/meb yum install -y libcgroup-tools touch meb.log tail -fn0 meb.log | awk '/Scanned log up to lsn/ { print $0 ; system("echo \"253:0 1000000\" > /sys/fs/cgroup/blkio/meb/blkio.throttle.write_bps_device") }' tail -fn0 meb.log | awk '/is locked for backup/ {print $0 ; system("sh /root/run_load.sh") }' tail -fn0 meb.log | awk '/Completed reading of all/ { print $0 ; system("pkill -9 sysbench") }' cat /root/run_load.sh sysbench oltp_update_index --mysql_storage_engine=innodb --table-size=100000 --tables=1 --mysql-db=test --mysql-port=3306 --mysql-host=127.0.0.1 --mysql-user=root --mysql-password=Actionsky!888 --threads=10 --time=100 --report-interval=1 --events=0 --db-driver=mysql run 2. start mysqlbackup witch cgexec rm -rf backup && cgexec -g blkio:meb ./mysqlbackup --backup-dir=backup --trace=3 backup > meb.log 2>&1 3. apply log and copy back ./mysqlbackup --backup-dir=backup --trace=3 apply-log completed OK! not stop mysqld service ./mysqlbackup --backup-dir=backup --trace=3 --force copy-back INFO: Connection to server is not available for query 'SHOW ENGINE INNODB STATUS' INFO: Cannot get redo log flush status why get redo log flush status failed, mysqld service was running? completed OK! =========backup complete, output table count and engine status============ --- LOG --- Log sequence number 3564527043 Log buffer assigned up to 3564527043 Log buffer completed up to 3564527043 Log written up to 3564527043 Log flushed up to 3564527043 Added dirty pages up to 3564527043 Pages flushed up to 3564527043 Last checkpoint at 3564527043 4425739 log i/o's done, 0.00 log i/o's/second ... ouput truncation 1 row in set (0.00 sec) mysql> select count(*) from sbtest1; +----------+ | count(*) | +----------+ | 2408610 | +----------+ 1 row in set (0.33 sec) =========force copy back complete, output table count and engine status============ mysql> select count(*) from sbtest1; ERROR 2013 (HY000): Lost connection to MySQL server during query mysql> select count(*) from sbtest1; ERROR 2006 (HY000): MySQL server has gone away No connection. Trying to reconnect... ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (111) ERROR: Can't connect to the server mysql> select count(*) from sbtest1; No connection. Trying to reconnect... Connection id: 8 Current database: test +----------+ | count(*) | +----------+ | 1903810 | +----------+ 1 row in set (0.87 sec) mysql> show engine innodb status\G *************************** 1. row *************************** ......output truncation.... --- LOG --- Log sequence number 3277356247 Log buffer assigned up to 3277356247 Log buffer completed up to 3277356247 Log written up to 3277356247 Log flushed up to 3277356247 Added dirty pages up to 3277356247 Pages flushed up to 3277356247 Last checkpoint at 3277356247 50 log i/o's done, 2.47 log i/o's/second