MySQL Bugs: #73155: InnoDB: index "CLUST_IND" of table "SYS

Bug #73155	InnoDB: index "CLUST_IND" of table "SYS_TABLES"
Submitted:	1 Jul 2014 6:40	Modified:	27 Apr 2016 12:02
Reporter:	Max Gao	Email Updates:
Status:	Closed	Impact on me:	None
Category:	MySQL Server: InnoDB storage engine	Severity:	S2 (Serious)
Version:	5.6.19	OS:	Linux (Centos 6.x)
Assigned to:		CPU Architecture:	Any

Description:
like the bug #69898

i want to perform a upgrade to the mysql today,
the version is from 5.6.10 to 5.6.19,
so i try it at the slave server first.
using rpm -Uvh rpm, after that use /etc/init.d/mysql start
the server crash and error logs below.

downgrade back to 5.6.10 and every thing works fine.

please some one check it ?

140701 14:08:13 mysqld_safe Starting mysqld daemon with databases from /home/mysql
2014-07-01 14:08:14 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestamp server option (see documentation for more details).
2014-07-01 14:08:14 11224 [Warning] Buffered warning: Using unique option prefix max_connection instead of max_connections is deprecated and will be removed in a future release. Please use the full name instead.

2014-07-01 14:08:14 11224 [Note] Plugin 'FEDERATED' is disabled.
2014-07-01 14:08:14 11224 [Note] InnoDB: Using atomics to ref count buffer pool pages
2014-07-01 14:08:14 11224 [Note] InnoDB: The InnoDB memory heap is disabled
2014-07-01 14:08:14 11224 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
2014-07-01 14:08:14 11224 [Note] InnoDB: Compressed tables use zlib 1.2.3
2014-07-01 14:08:14 11224 [Note] InnoDB: Using Linux native AIO
2014-07-01 14:08:14 11224 [Note] InnoDB: Not using CPU crc32 instructions
2014-07-01 14:08:14 11224 [Note] InnoDB: Initializing buffer pool, size = 4.0G
2014-07-01 14:08:15 11224 [Note] InnoDB: Completed initialization of buffer pool
2014-07-01 14:08:15 11224 [Note] InnoDB: Highest supported file format is Barracuda.
2014-07-01 14:08:15 7fd24b04c720  InnoDB: Error: a record lock wait happens in a dictionary operation!
InnoDB: index "CLUST_IND" of table "SYS_TABLES".
InnoDB: Submit a detailed bug report to http://bugs.mysql.com
2014-07-01 14:08:15 7fd24b04c720  InnoDB: Assertion failure in thread 140541178464032 in file lock0wait.cc line 297
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: http://dev.mysql.com/doc/refman/5.6/en/forcing-innodb-recovery.html
InnoDB: about forcing recovery.
06:08:15 UTC - mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed, 
something is definitely wrong and this may fail.

key_buffer_size=4294967296
read_buffer_size=4194304
max_used_connections=0
max_threads=512
thread_count=0
connection_count=0
It is possible that mysqld could use up to 
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 7347048 K  bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0x0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0 thread_stack 0x40000
/usr/sbin/mysqld(my_print_stacktrace+0x35)[0x8d68f5]
/usr/sbin/mysqld(handle_fatal_signal+0x4a4)[0x663224]
/lib64/libpthread.so.0(+0xf710)[0x7fd24ac2e710]
/lib64/libc.so.6(gsignal+0x35)[0x7fd249ada925]
/lib64/libc.so.6(abort+0x175)[0x7fd249adc105]
/usr/sbin/mysqld[0x927725]
/usr/sbin/mysqld[0x95c934]
/usr/sbin/mysqld[0x95cdf5]
/usr/sbin/mysqld[0x983af8]
/usr/sbin/mysqld[0xa5a04d]
/usr/sbin/mysqld[0x92e10f]
/usr/sbin/mysqld[0x9ac3b1]
/usr/sbin/mysqld[0x8f318d]
/usr/sbin/mysqld(_Z24ha_initialize_handlertonP13st_plugin_int+0x48)[0x5a7ac8]
/usr/sbin/mysqld[0x6ea751]
/usr/sbin/mysqld(_Z11plugin_initPiPPci+0xb76)[0x6ee636]
/usr/sbin/mysqld[0x59a7b8]
/usr/sbin/mysqld(_Z11mysqld_mainiPPc+0x455)[0x59f845]
/lib64/libc.so.6(__libc_start_main+0xfd)[0x7fd249ac6d1d]
/usr/sbin/mysqld[0x591ac9]
The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you find out what is causing the crash.
140701 14:08:15 mysqld_safe mysqld from pid file /home/mysql/localhost.localdomain.pid ended

How to repeat:
upgrade , start , fail
downgrade , start , fine , query / insert works fine too.

We have the same situation, we upgraded a MySQL server 5.6.16 to 5.6.19 and it won't start after the upgrade. Rolling back to 5.6.16 (rpm installation) and it works fine. Our OS is Red Hat Enterprise Linux 6.4.

Error log:

InnoDB: index "SYS_TABLESPACES_SPACE" of table "SYS_TABLESPACES".
InnoDB: Submit a detailed bug report to http://bugs.mysql.com
2014-07-01 23:49:02 7f85fc4fc720  InnoDB: Assertion failure in thread 140213440464672 in file lock0wait.cc line 297
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: http://dev.mysql.com/doc/refman/5.6/en/forcing-innodb-recovery.html
InnoDB: about forcing recovery.
21:49:02 UTC - mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.

key_buffer_size=536870912
read_buffer_size=131072
max_used_connections=0
max_threads=130
thread_count=0
connection_count=0
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 575989 K  bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0x0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0 thread_stack 0x40000
/usr/sbin/mysqld(my_print_stacktrace+0x35)[0x8d68f5]
/usr/sbin/mysqld(handle_fatal_signal+0x4a4)[0x663224]
/lib64/libpthread.so.0(+0xf710)[0x7f85fdaff710]
/lib64/libc.so.6(gsignal+0x35)[0x7f85fc7a9925]
/lib64/libc.so.6(abort+0x175)[0x7f85fc7ab105]
/usr/sbin/mysqld[0x927725]
/usr/sbin/mysqld[0x95c934]
/usr/sbin/mysqld[0x95cdf5]
/usr/sbin/mysqld[0x983af8]
/usr/sbin/mysqld[0xa5a04d]
/usr/sbin/mysqld[0x92e10f]
/usr/sbin/mysqld[0x9ac3b1]
/usr/sbin/mysqld[0x8f318d]
/usr/sbin/mysqld(_Z24ha_initialize_handlertonP13st_plugin_int+0x48)[0x5a7ac8]
/usr/sbin/mysqld[0x6ea751]
/usr/sbin/mysqld(_Z11plugin_initPiPPci+0xb76)[0x6ee636]
/usr/sbin/mysqld[0x59a7b8]
/usr/sbin/mysqld(_Z11mysqld_mainiPPc+0x455)[0x59f845]
/lib64/libc.so.6(__libc_start_main+0xfd)[0x7f85fc795d1d]
/usr/sbin/mysqld[0x591ac9]
The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you find out what is causing the crash.

it seems serious for the one who want to upgrade.

Thank you for the report.

I think we need information which Marko Makela requested in bug #69898:

The stack trace that you posted did not contain function names for InnoDB
functions, so we do not know which operation triggered the assertion failure.

Would it be possible to rerun with core dumps enabled, and to obtain a stack
trace? With gdb, it should be something like this:

gdb /path/to/mysqld core
(gdb) bt
(gdb) quit

No feedback was provided for this bug for over a month, so it is
being suspended automatically. If you are able to provide the
information that was originally requested, please do so and change
the status of the bug back to "Open".

Thank you for the report.
Verifying based on Developer's confirmation.

Thanks,
Umesh

Based on discussion with Sven and Thirunarayanan, this is not a replication bug. Therefore changing it to InnoDB.

Posted by developer:
 
Based on discussion with Sven and Thirunarayanan, this is not a replication bug. Therefore changing it to InnoDB.

Posted by developer:
 
This bug was addressed bug the fix for Bug #18285007:

"Noted in 5.6.22, 5.7.6 changelogs.

Copying InnoDB tables containing full-text columns from Windows to
Linux caused a server exit on Linux during full-text index
initialization."

Thank you for the bug report.