Bug #113706 InnoDB : Corruption of an index tree which stopped mysqld service
Submitted: 22 Jan 11:46 Modified: 22 Jan 13:20
Reporter: CEOVISION CEOVISION Email Updates:
Status: Can't repeat Impact on me:
None 
Category:MySQL Server: Errors Severity:S3 (Non-critical)
Version:mysql Ver 8.0.35 for Linux on 86_64 OS:Linux (Centos 7 Linux )
Assigned to: CPU Architecture:x86 (x86_64)

[22 Jan 11:46] CEOVISION CEOVISION
Description:
Hello,
Following an index corruption on one of the tables of a database, mysqld service suddenly stopped (mysqld got signal 6). We don't know whether this bug could be due to a bug in our mysqld service configuration or to an external problem (memory corruption, data replication...) but, for the moment, we haven't detected any such problems. 
Could you tell us if the problem could be linked to our configuration file or, in the case of an external problem, what could be the causes?

Here are some of the mysql server logs:

2023-12-28T14:46:45.885241Z 0 [Warning] [MY-011070] [Server] 'Disabling symbolic links using --skip-symbolic-links (or equivalent) is the default. Consider not using this option as it' is deprecated and will be removed in a future release.
2023-12-28T14:46:45.885276Z 0 [Warning] [MY-011070] [Server] 'Disabling symbolic links using --skip-symbolic-links (or equivalent) is the default. Consider not using this option as it' is deprecated and will be removed in a future release.
2023-12-28T14:46:45.887060Z 0 [System] [MY-010116] [Server] /usr/sbin/mysqld (mysqld 8.0.35) starting as process 20298
2023-12-28T14:46:45.904757Z 0 [Warning] [MY-013907] [InnoDB] Deprecated configuration parameters innodb_log_file_size and/or innodb_log_files_in_group have been used to compute innodb_redo_log_capacity=209715200. Please use innodb_redo_log_capacity instead.
2023-12-28T14:46:45.906660Z 1 [System] [MY-013576] [InnoDB] InnoDB initialization has started.
2023-12-28T14:46:47.901279Z 1 [System] [MY-013577] [InnoDB] InnoDB initialization has ended.
2023-12-28T14:46:48.919133Z 0 [Warning] [MY-013129] [Server] A message intended for a client cannot be sent there as no client-session is attached. Therefore, we're sending the information to the error-log instead: MY-001287 - 'validate password plugin' is deprecated and will be removed in a future release. Please use validate_password component instead
2023-12-28T14:46:49.122781Z 0 [ERROR] [MY-011853] [InnoDB] Corruption of an index tree: table `xxx`.`xxxxxx` index `xxxxxx`, father ptr page no 684, child page no 682
PHYSICAL RECORD: n_fields 5; compact format; info bits 0
 0: len 4; hex 00031b12; asc     ;;
 1: len 4; hex 6e6f6465; asc node;;
 2: len 1; hex 80; asc  ;;
 3: len 4; hex 00000000; asc     ;;
 4: len 3; hex 756e64; asc und;;
PHYSICAL RECORD: n_fields 6; compact format; info bits 0
 0: len 4; hex 00031b12; asc     ;;
 1: len 4; hex 6e6f6465; asc node;;
 2: len 1; hex 80; asc  ;;
 3: len 4; hex 00000000; asc     ;;
 4: len 3; hex 756e64; asc und;;
 5: len 4; hex 000002ac; asc     ;;
2023-12-28T14:46:49.124197Z 0 [ERROR] [MY-011854] [InnoDB] [FATAL] You should dump + drop + reimport the table to fix the corruption. If the crash happens at database startup. Please refer to http://dev.mysql.com/doc/refman/8.0/en/forcing-innodb-recovery.html for information about forcing recovery. Then dump + drop + reimport.
2023-12-28T14:46:49.124223Z 0 [ERROR] [MY-013183] [InnoDB] Assertion failure: btr0btr.cc:700:ib::fatal triggered thread 139866115671808
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: http://dev.mysql.com/doc/refman/8.0/en/forcing-innodb-recovery.html
InnoDB: about forcing recovery.
2023-12-28T14:46:49Z UTC - mysqld got signal 6 ;
Most likely, you have hit a bug, but this error can also be caused by malfunctioning hardware.
BuildID[sha1]=b62ed69988655f6fa72a332a71e62a40867984b7
Thread pointer: 0x7f34c40008c0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 7f351e23fab0 thread_stack 0x100000
/usr/sbin/mysqld(my_print_stacktrace(unsigned char const*, unsigned long)+0x3d) [0x211a20d]
/usr/sbin/mysqld(print_fatal_signal(int)+0x37f) [0xfd8b6f]
/usr/sbin/mysqld(my_server_abort()+0x7e) [0xfd8cbe]
/usr/sbin/mysqld(my_abort()+0xa) [0x21143da]
/usr/sbin/mysqld(ut_dbg_assertion_failed(char const*, char const*, unsigned long)+0x31f) [0x2334fdf]
/usr/sbin/mysqld() [0x233799f]
/usr/sbin/mysqld() [0x2362c96]
/usr/sbin/mysqld(btr_compress(btr_cur_t*, bool, mtr_t*)+0x9c2) [0x2366032]
/usr/sbin/mysqld(btr_cur_pessimistic_delete(dberr_t*, bool, btr_cur_t*, unsigned int, bool, unsigned long, unsigned long, unsigned long, mtr_t*, btr_pcur_t*, purge_node_t*)+0x223) [0x2373463]
/usr/sbin/mysqld() [0x24ec2be]
/usr/sbin/mysqld(row_undo_mod(undo_node_t*, que_thr_t*)+0xb7d) [0x24edbcd]
/usr/sbin/mysqld(row_undo_step(que_thr_t*)+0x65) [0x22c4555]
/usr/sbin/mysqld(que_run_threads(que_thr_t*)+0x440) [0x226f1a0]
/usr/sbin/mysqld(trx_rollback_or_clean_recovered(bool)+0x5d3) [0x2316d53]
/usr/sbin/mysqld(trx_recovery_rollback(THD*)+0x533) [0x2317723]
/usr/sbin/mysqld(trx_recovery_rollback_thread()+0x1a) [0x23179aa]
/usr/sbin/mysqld(void Detached_thread::operator()(void (*&&)())+0xc2) [0x22258a2]
/usr/sbin/mysqld() [0x2adf914]
/lib64/libpthread.so.0(+0x7ea5) [0x7f352dd56ea5]
/lib64/libc.so.6(clone+0x6d) [0x7f352c370b0d]

Trying to get some variables.
Some pointers may be invalid and cause the dump to abort.
Query (0): Connection ID (thread ID): 0
Status: NOT_KILLED

The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you find out what is causing the crash.

Our mysqld configuration file (/etc/my.cnf):

# For advice on how to change settings please see
# http://dev.mysql.com/doc/refman/5.7/en/server-configuration-defaults.html

[mysqld]
max_allowed_packet = 16M
user=mysql
symbolic-links=0
tmp_table_size = 50M
#query_cache_size = 50M
#query_cache_limit = 20M
#query_cache_type = 1
max_connections = 150
max_user_connections = 150
thread_cache_size = 286
interactive_timeout = 1800
wait_timeout = 1800
connect_timeout = 10
sort_buffer_size = 3M
join_buffer_size = 6M
read_buffer_size = 2M
innodb_buffer_pool_size=20M
max_heap_table_size=50M
#table_cache = 1024
#key_buffer = 128M

#
# Remove leading # and set to the amount of RAM for the most important data
# cache in MySQL. Start at 70% of total RAM for dedicated server, else 10%.
# innodb_buffer_pool_size = 128M
#
# Remove leading # to turn on a very important data integrity option: logging
# changes to the binary log between backups.
# log_bin
#
# Remove leading # to set options mainly useful for reporting servers.
# The server defaults are faster for transactions and fast SELECTs.
# Adjust sizes as needed, experiment to find the optimal values.
# join_buffer_size = 128M
# sort_buffer_size = 2M
# read_rnd_buffer_size = 2M
datadir=/var/lib/mysql
socket=/var/lib/mysql/mysql.sock

# Disabling symbolic-links is recommended to prevent assorted security risks
symbolic-links=0

log-error=/var/log/mysql/mysqld.log
pid-file=/var/run/mysqld/mysqld.pid
#skip-innodb
default-storage-engine = innodb
innodb_buffer_pool_size = 200M
innodb_log_file_size = 100M
innodb_flush_method = O_DIRECT
innodb_file_per_table = 1
innodb_flush_log_at_trx_commit = 2
innodb_log_buffer_size = 4M
#innodb_additional_mem_pool_size = 20M
#num cpu's/cores *2 is a good base line for innodb_thread_concurrency
innodb_thread_concurrency = 4

How to repeat:
- restart innodb using force-recovery
- table recreation : duplicate old table schema and add data 
- delete old table
- restart mysqld service
[22 Jan 13:20] MySQL Verification Team
Hi Mr. Ceovision,

Thank you for your bug report.

However, let us inform you that this is a forum for reports with  a fully repeatable test cases. Hence, we must have a test case that will result in the same error or problem that you have encountered.

Only when we have such a test case, written fully and only as a set of SQL statements, only then we might be able to repeat it . If  we repeat it, then we can verify the bug report and put it in our internal bugs database.

Next, we do not accept repeatable test cases which include configuration elements that are deprecated or no longer valid for the version / release for which the problem is reported.

Hence, we can not do anything with your report, until we receive full information, as it is disclosed above.

Can't repeat.