Bug #91036 mysql 8.0.11 innodb corrupt after sysbench failed
Submitted: 27 May 2018 4:45 Modified: 9 Jul 2018 16:57
Reporter: li zhenxu Email Updates:
Status: Can't repeat Impact on me:
None 
Category:MySQL Server: InnoDB storage engine Severity:S3 (Non-critical)
Version:8.0.11 OS:Any (3.10.0-514.26.2.el7.x86_64)
Assigned to: CPU Architecture:x86

[27 May 2018 4:45] li zhenxu
Description:
Mysqld crash after sysbench test。 Test command as  fellow:

/root/sysbench-1.0/src/sysbench oltp_read_write.lua --mysql-host=127.0.0.1 --mysql-port=3306 --mysql-db=enmotech --mysql-user=root --mysql-password=enmotech --table_size=1000000 --tables=100  --threads=50  --report-interval=10 --time=300 prepare

The error.log:

2018-05-27T04:35:29.643955Z 0 [Warning] [MY-011070] [Server] 'Disabling symbolic links using --skip-symbolic-links (or equivalent) is the default. Consider not using this option as it' is deprecated and will be removed in a future release.
2018-05-27T04:35:29.644184Z 0 [System] [MY-010116] [Server] mysqld (mysqld 8.0.11) starting as process 19586
2018-05-27T04:35:31.014343Z 1 [ERROR] [MY-000000] [InnoDB] InnoDB: Assertion failure: buf0rea.cc:151
InnoDB: thread 139869132433152
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: http://dev.mysql.com/doc/refman/8.0/en/forcing-innodb-recovery.html
InnoDB: about forcing recovery.
04:35:31 UTC - mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
Attempting to collect some information that could help diagnose the problem.
As this is a crash and something is definitely wrong, the information
collection process might fail.

key_buffer_size=536870912
read_buffer_size=2097152
max_used_connections=0
max_threads=1500
thread_count=1
connection_count=0
It is possible that mysqld could use up to 
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 101916834 K  bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0x42c4b50
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 7f35d1f41dc8 thread_stack 0x80000
mysqld(my_print_stacktrace(unsigned char*, unsigned long)+0x2e) [0x19fcc8e]
mysqld(handle_fatal_signal+0x413) [0xccc863]
/lib64/libpthread.so.0(+0xf370) [0x7f3604949370]
/lib64/libc.so.6(gsignal+0x37) [0x7f3602ce51d7]
/lib64/libc.so.6(abort+0x148) [0x7f3602ce68c8]
mysqld() [0xac6ea8]
mysqld() [0x1d9ac2b]
mysqld(buf_read_page(page_id_t const&, page_size_t const&)+0x37) [0x1d9af27]
mysqld(buf_page_get_gen(page_id_t const&, page_size_t const&, unsigned long, buf_block_t*, unsigned long, char const*, unsigned long, mtr_t*, bool)+0x4c4) [0x1d6c584]
mysqld(trx_undo_lists_init(trx_rseg_t*)+0x934) [0x1d12664]
mysqld(trx_rseg_mem_create(unsigned long, unsigned int, unsigned int, page_size_t const&, std::priority_queue<TrxUndoRsegs, std::vector<TrxUndoRsegs, ut_allocator<TrxUndoRsegs> >, TrxUndoRsegs>*, mtr_t*)+0x3fb) [0x1cf892b]
mysqld(trx_rsegs_init(std::priority_queue<TrxUndoRsegs, std::vector<TrxUndoRsegs, ut_allocator<TrxUndoRsegs> >, TrxUndoRsegs>*)+0x3c4) [0x1cfa194]
mysqld(trx_sys_init_at_db_start()+0xe64) [0x1d005c4]
mysqld(srv_start(bool, std::string const&)+0x3335) [0x1cc8815]
mysqld() [0x1b6fd25]
mysqld(dd::bootstrap::DDSE_dict_init(THD*, dict_init_mode_t, unsigned int)+0x77) [0x199cc97]
mysqld(dd::upgrade_57::do_pre_checks_and_initialize_dd(THD*)+0x2d6) [0x1994c76]
mysqld() [0xd70f06]
mysqld() [0x1aa5f5f]
/lib64/libpthread.so.0(+0x7dc5) [0x7f3604941dc5]
/lib64/libc.so.6(clone+0x6d) [0x7f3602da776d]

Trying to get some variables.
Some pointers may be invalid and cause the dump to abort.
Query (0): is an invalid pointer
Connection ID (thread ID): 1
Status: NOT_KILLED

The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you find out what is causing the crash.

the mysql instance is not startup after add fellow parameter :
innodb_purge_thread=0
innodb_force_recovery = 6

The error  is still the same。

How to repeat:
not repeat!
[9 Jul 2018 16:57] MySQL Verification Team
Hi,

I have run your test on 8.0.11 like this:

./sysbench oltp_read_write.lua --mysql-host=127.0.0.1 --mysql-port=3306 --mysql-db=test --mysql-user=myuser --mysql-password=mypassword  --table_size=1000000 --tables=100  --threads=50  --report-interval=10 --time=300 prepare

And everything ran just fine .... these are the last lines:

Inserting 1000000 records into 'sbtest75'
Creating a secondary index on 'sbtest37'...
Creating a secondary index on 'sbtest3'...
Creating table 'sbtest87'...
Inserting 1000000 records into 'sbtest87'
Creating table 'sbtest53'...
Inserting 1000000 records into 'sbtest53'
Creating a secondary index on 'sbtest39'...
Creating table 'sbtest89'...
Inserting 1000000 records into 'sbtest89'
Creating a secondary index on 'sbtest6'...
Creating table 'sbtest56'...
Inserting 1000000 records into 'sbtest56'
Creating a secondary index on 'sbtest43'...
Creating table 'sbtest93'...
Inserting 1000000 records into 'sbtest93'
Creating a secondary index on 'sbtest50'...
Creating table 'sbtest100'...
Inserting 1000000 records into 'sbtest100'
Creating a secondary index on 'sbtest40'...
Creating table 'sbtest90'...
Inserting 1000000 records into 'sbtest90'
Creating a secondary index on 'sbtest24'...
Creating table 'sbtest74'...
Inserting 1000000 records into 'sbtest74'
Creating a secondary index on 'sbtest17'...
Creating table 'sbtest67'...
Inserting 1000000 records into 'sbtest67'
Creating a secondary index on 'sbtest69'...
Creating a secondary index on 'sbtest83'...
Creating a secondary index on 'sbtest55'...
Creating a secondary index on 'sbtest77'...
Creating a secondary index on 'sbtest65'...
Creating a secondary index on 'sbtest62'...
Creating a secondary index on 'sbtest84'...
Creating a secondary index on 'sbtest91'...
Creating a secondary index on 'sbtest96'...
Creating a secondary index on 'sbtest76'...
Creating a secondary index on 'sbtest54'...
Creating a secondary index on 'sbtest88'...
Creating a secondary index on 'sbtest85'...
Creating a secondary index on 'sbtest60'...
Creating a secondary index on 'sbtest66'...
Creating a secondary index on 'sbtest81'...
Creating a secondary index on 'sbtest94'...
Creating a secondary index on 'sbtest89'...
Creating a secondary index on 'sbtest95'...
Creating a secondary index on 'sbtest80'...
Creating a secondary index on 'sbtest86'...
Creating a secondary index on 'sbtest70'...
Creating a secondary index on 'sbtest79'...
Creating a secondary index on 'sbtest97'...
Creating a secondary index on 'sbtest59'...
Creating a secondary index on 'sbtest78'...
Creating a secondary index on 'sbtest92'...
Creating a secondary index on 'sbtest57'...
Creating a secondary index on 'sbtest61'...
Creating a secondary index on 'sbtest73'...
Creating a secondary index on 'sbtest72'...
Creating a secondary index on 'sbtest58'...
Creating a secondary index on 'sbtest75'...
Creating a secondary index on 'sbtest63'...
Creating a secondary index on 'sbtest51'...
Creating a secondary index on 'sbtest64'...
Creating a secondary index on 'sbtest93'...
Creating a secondary index on 'sbtest99'...
Creating a secondary index on 'sbtest68'...
Creating a secondary index on 'sbtest82'...
Creating a secondary index on 'sbtest74'...
Creating a secondary index on 'sbtest98'...
Creating a secondary index on 'sbtest52'...
Creating a secondary index on 'sbtest87'...
Creating a secondary index on 'sbtest90'...
Creating a secondary index on 'sbtest56'...
Creating a secondary index on 'sbtest67'...
Creating a secondary index on 'sbtest71'...
Creating a secondary index on 'sbtest100'...
Creating a secondary index on 'sbtest53'...
[10 Jul 2018 13:50] Roger li
hi, Sinisa Milivojevic,
     
      my mysql is mgr. as fellow:

mysql> select * from performance_schema.replication_group_members;
+---------------------------+--------------------------------------+-------------+-------------+--------------+-------------+----------------+
| CHANNEL_NAME              | MEMBER_ID                            | MEMBER_HOST | MEMBER_PORT | MEMBER_STATE | MEMBER_ROLE | MEMBER_VERSION |
+---------------------------+--------------------------------------+-------------+-------------+--------------+-------------+----------------+
| group_replication_applier | 25d0899d-5eff-11e8-a6bf-00163e007fde | enmodb2     |        3306 | ONLINE       | PRIMARY     | 8.0.11         |
| group_replication_applier | 4f89cd14-5eff-11e8-adf8-00163e004de6 | enmodb3     |        3306 | ONLINE       | PRIMARY     | 8.0.11         |
| group_replication_applier | b3dfe8d1-64b2-11e8-be29-00163e005628 | enmodb1     |        3306 | ONLINE       | PRIMARY     | 8.0.11         |
+---------------------------+--------------------------------------+-------------+-------------+--------------+-------------+----------------+
3 rows in set (0.01 sec)

I haved rebuild the database.
[10 Jul 2018 14:53] MySQL Verification Team
Hi,

What do you mean that MySQL is manager ???

Is it master or slave ???  Hence , does this crash happens on the master or slave ??

Or are you using MySQL GR, also known as InnoDB Cluster. In that case, please state which machine exactly crashes in the Cluster. Draw the topology of the GR and let us know which MySQL exactly crashes on sysbench  oltp read-write test.