Description:
Hello,
Following an index corruption on one of the tables of a database, mysqld service suddenly stopped (mysqld got signal 6). We don't know whether this bug could be due to a bug in our mysqld service configuration or to an external problem (memory corruption, data replication...) but, for the moment, we haven't detected any such problems.
Could you tell us if the problem could be linked to our configuration file or, in the case of an external problem, what could be the causes?
Here are some of the mysql server logs:
2023-12-28T14:46:45.885241Z 0 [Warning] [MY-011070] [Server] 'Disabling symbolic links using --skip-symbolic-links (or equivalent) is the default. Consider not using this option as it' is deprecated and will be removed in a future release.
2023-12-28T14:46:45.885276Z 0 [Warning] [MY-011070] [Server] 'Disabling symbolic links using --skip-symbolic-links (or equivalent) is the default. Consider not using this option as it' is deprecated and will be removed in a future release.
2023-12-28T14:46:45.887060Z 0 [System] [MY-010116] [Server] /usr/sbin/mysqld (mysqld 8.0.35) starting as process 20298
2023-12-28T14:46:45.904757Z 0 [Warning] [MY-013907] [InnoDB] Deprecated configuration parameters innodb_log_file_size and/or innodb_log_files_in_group have been used to compute innodb_redo_log_capacity=209715200. Please use innodb_redo_log_capacity instead.
2023-12-28T14:46:45.906660Z 1 [System] [MY-013576] [InnoDB] InnoDB initialization has started.
2023-12-28T14:46:47.901279Z 1 [System] [MY-013577] [InnoDB] InnoDB initialization has ended.
2023-12-28T14:46:48.919133Z 0 [Warning] [MY-013129] [Server] A message intended for a client cannot be sent there as no client-session is attached. Therefore, we're sending the information to the error-log instead: MY-001287 - 'validate password plugin' is deprecated and will be removed in a future release. Please use validate_password component instead
2023-12-28T14:46:49.122781Z 0 [ERROR] [MY-011853] [InnoDB] Corruption of an index tree: table `xxx`.`xxxxxx` index `xxxxxx`, father ptr page no 684, child page no 682
PHYSICAL RECORD: n_fields 5; compact format; info bits 0
0: len 4; hex 00031b12; asc ;;
1: len 4; hex 6e6f6465; asc node;;
2: len 1; hex 80; asc ;;
3: len 4; hex 00000000; asc ;;
4: len 3; hex 756e64; asc und;;
PHYSICAL RECORD: n_fields 6; compact format; info bits 0
0: len 4; hex 00031b12; asc ;;
1: len 4; hex 6e6f6465; asc node;;
2: len 1; hex 80; asc ;;
3: len 4; hex 00000000; asc ;;
4: len 3; hex 756e64; asc und;;
5: len 4; hex 000002ac; asc ;;
2023-12-28T14:46:49.124197Z 0 [ERROR] [MY-011854] [InnoDB] [FATAL] You should dump + drop + reimport the table to fix the corruption. If the crash happens at database startup. Please refer to http://dev.mysql.com/doc/refman/8.0/en/forcing-innodb-recovery.html for information about forcing recovery. Then dump + drop + reimport.
2023-12-28T14:46:49.124223Z 0 [ERROR] [MY-013183] [InnoDB] Assertion failure: btr0btr.cc:700:ib::fatal triggered thread 139866115671808
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: http://dev.mysql.com/doc/refman/8.0/en/forcing-innodb-recovery.html
InnoDB: about forcing recovery.
2023-12-28T14:46:49Z UTC - mysqld got signal 6 ;
Most likely, you have hit a bug, but this error can also be caused by malfunctioning hardware.
BuildID[sha1]=b62ed69988655f6fa72a332a71e62a40867984b7
Thread pointer: 0x7f34c40008c0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 7f351e23fab0 thread_stack 0x100000
/usr/sbin/mysqld(my_print_stacktrace(unsigned char const*, unsigned long)+0x3d) [0x211a20d]
/usr/sbin/mysqld(print_fatal_signal(int)+0x37f) [0xfd8b6f]
/usr/sbin/mysqld(my_server_abort()+0x7e) [0xfd8cbe]
/usr/sbin/mysqld(my_abort()+0xa) [0x21143da]
/usr/sbin/mysqld(ut_dbg_assertion_failed(char const*, char const*, unsigned long)+0x31f) [0x2334fdf]
/usr/sbin/mysqld() [0x233799f]
/usr/sbin/mysqld() [0x2362c96]
/usr/sbin/mysqld(btr_compress(btr_cur_t*, bool, mtr_t*)+0x9c2) [0x2366032]
/usr/sbin/mysqld(btr_cur_pessimistic_delete(dberr_t*, bool, btr_cur_t*, unsigned int, bool, unsigned long, unsigned long, unsigned long, mtr_t*, btr_pcur_t*, purge_node_t*)+0x223) [0x2373463]
/usr/sbin/mysqld() [0x24ec2be]
/usr/sbin/mysqld(row_undo_mod(undo_node_t*, que_thr_t*)+0xb7d) [0x24edbcd]
/usr/sbin/mysqld(row_undo_step(que_thr_t*)+0x65) [0x22c4555]
/usr/sbin/mysqld(que_run_threads(que_thr_t*)+0x440) [0x226f1a0]
/usr/sbin/mysqld(trx_rollback_or_clean_recovered(bool)+0x5d3) [0x2316d53]
/usr/sbin/mysqld(trx_recovery_rollback(THD*)+0x533) [0x2317723]
/usr/sbin/mysqld(trx_recovery_rollback_thread()+0x1a) [0x23179aa]
/usr/sbin/mysqld(void Detached_thread::operator()(void (*&&)())+0xc2) [0x22258a2]
/usr/sbin/mysqld() [0x2adf914]
/lib64/libpthread.so.0(+0x7ea5) [0x7f352dd56ea5]
/lib64/libc.so.6(clone+0x6d) [0x7f352c370b0d]
Trying to get some variables.
Some pointers may be invalid and cause the dump to abort.
Query (0): Connection ID (thread ID): 0
Status: NOT_KILLED
The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you find out what is causing the crash.
Our mysqld configuration file (/etc/my.cnf):
# For advice on how to change settings please see
# http://dev.mysql.com/doc/refman/5.7/en/server-configuration-defaults.html
[mysqld]
max_allowed_packet = 16M
user=mysql
symbolic-links=0
tmp_table_size = 50M
#query_cache_size = 50M
#query_cache_limit = 20M
#query_cache_type = 1
max_connections = 150
max_user_connections = 150
thread_cache_size = 286
interactive_timeout = 1800
wait_timeout = 1800
connect_timeout = 10
sort_buffer_size = 3M
join_buffer_size = 6M
read_buffer_size = 2M
innodb_buffer_pool_size=20M
max_heap_table_size=50M
#table_cache = 1024
#key_buffer = 128M
#
# Remove leading # and set to the amount of RAM for the most important data
# cache in MySQL. Start at 70% of total RAM for dedicated server, else 10%.
# innodb_buffer_pool_size = 128M
#
# Remove leading # to turn on a very important data integrity option: logging
# changes to the binary log between backups.
# log_bin
#
# Remove leading # to set options mainly useful for reporting servers.
# The server defaults are faster for transactions and fast SELECTs.
# Adjust sizes as needed, experiment to find the optimal values.
# join_buffer_size = 128M
# sort_buffer_size = 2M
# read_rnd_buffer_size = 2M
datadir=/var/lib/mysql
socket=/var/lib/mysql/mysql.sock
# Disabling symbolic-links is recommended to prevent assorted security risks
symbolic-links=0
log-error=/var/log/mysql/mysqld.log
pid-file=/var/run/mysqld/mysqld.pid
#skip-innodb
default-storage-engine = innodb
innodb_buffer_pool_size = 200M
innodb_log_file_size = 100M
innodb_flush_method = O_DIRECT
innodb_file_per_table = 1
innodb_flush_log_at_trx_commit = 2
innodb_log_buffer_size = 4M
#innodb_additional_mem_pool_size = 20M
#num cpu's/cores *2 is a good base line for innodb_thread_concurrency
innodb_thread_concurrency = 4
How to repeat:
- restart innodb using force-recovery
- table recreation : duplicate old table schema and add data
- delete old table
- restart mysqld service
