| Bug #72080 | truncate temporary table crash: !DICT_TF2_FLAG_IS_SET(table, DICT_TF2_TEMPORARY) | ||
|---|---|---|---|
| Submitted: | 19 Mar 2014 18:06 | Modified: | 23 Feb 2015 14:35 |
| Reporter: | Doug Warner | Email Updates: | |
| Status: | Closed | Impact on me: | |
| Category: | MySQL Server: InnoDB storage engine | Severity: | S1 (Critical) |
| Version: | 5.6.15 | OS: | Any |
| Assigned to: | CPU Architecture: | Any | |
| Tags: | DICT_TF2_TEMPORARY, temporary table, truncate | ||
[19 Mar 2014 18:06]
Doug Warner
[19 Mar 2014 18:07]
Doug Warner
Additional log files from another server: 2014-03-19 03:47:20 7fd3d4d37700 InnoDB: Assertion failure in thread 140547785455360 in file row0mysql.cc line 3336 InnoDB: Failing assertion: !DICT_TF2_FLAG_IS_SET(table, DICT_TF2_TEMPORARY) InnoDB: We intentionally generate a memory trap. InnoDB: Submit a detailed bug report to http://bugs.mysql.com. InnoDB: If you get repeated assertion failures or crashes, even InnoDB: immediately after the mysqld startup, there may be InnoDB: corruption in the InnoDB tablespace. Please refer to InnoDB: http://dev.mysql.com/doc/refman/5.6/en/forcing-innodb-recovery.html InnoDB: about forcing recovery. 07:47:20 UTC - mysqld got signal 6 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. key_buffer_size=268435456 read_buffer_size=16777216 max_used_connections=1713 max_threads=3000 thread_count=185 connection_count=185 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 74031136 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x1bf94490 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 7fd3d4d36e30 thread_stack 0x40000 /usr/sbin/mysqld(my_print_stacktrace+0x35)[0x8cdf65] /usr/sbin/mysqld(handle_fatal_signal+0x413)[0x6382f3] /lib/libpthread.so.0(+0x10280)[0x7fd5b2054280] /lib/libc.so.6(gsignal+0x35)[0x7fd5b16d4675] /lib/libc.so.6(abort+0x17a)[0x7fd5b16d591a] /usr/sbin/mysqld[0xa0fc7c] /usr/sbin/mysqld[0x981e0b] /usr/sbin/mysqld(_ZN22Sql_cmd_truncate_table14truncate_tableEP3THDP10TABLE_LIST+0x3a9)[0x84f209] /usr/sbin/mysqld(_ZN22Sql_cmd_truncate_table7executeEP3THD+0x5e)[0x84f2fe] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x2bb1)[0x6be431] /usr/sbin/mysqld(_Z11mysql_parseP3THDPcjP12Parser_state+0x398)[0x6c2188] /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcj+0x20a5)[0x6c42c5] /usr/sbin/mysqld(_Z24do_handle_one_connectionP3THD+0x1ed)[0x68bfcd] /usr/sbin/mysqld(handle_one_connection+0x42)[0x68c022] /usr/sbin/mysqld(pfs_spawn_thread+0x127)[0x90a597] /lib/libpthread.so.0(+0x7d56)[0x7fd5b204bd56] /lib/libc.so.6(clone+0x6d)[0x7fd5b1787efd] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (7fd107d08dd0): is an invalid pointer Connection ID (thread ID): 285352518 Status: NOT_KILLED 2014-03-19 08:11:45 7f9919ebc700 InnoDB: Assertion failure in thread 140295541606144 in file row0mysql.cc line 3336 InnoDB: Failing assertion: !DICT_TF2_FLAG_IS_SET(table, DICT_TF2_TEMPORARY) InnoDB: We intentionally generate a memory trap. InnoDB: Submit a detailed bug report to http://bugs.mysql.com. InnoDB: If you get repeated assertion failures or crashes, even InnoDB: immediately after the mysqld startup, there may be InnoDB: corruption in the InnoDB tablespace. Please refer to InnoDB: http://dev.mysql.com/doc/refman/5.6/en/forcing-innodb-recovery.html InnoDB: about forcing recovery. 12:11:45 UTC - mysqld got signal 6 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. key_buffer_size=268435456 read_buffer_size=16777216 max_used_connections=1338 max_threads=3000 thread_count=1338 connection_count=1338 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 74031136 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x2b3f4f30 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 7f9919ebbe30 thread_stack 0x40000 /usr/sbin/mysqld(my_print_stacktrace+0x35)[0x8cdf65] /usr/sbin/mysqld(handle_fatal_signal+0x413)[0x6382f3] /lib/libpthread.so.0(+0x10280)[0x7f9af2478280] /lib/libc.so.6(gsignal+0x35)[0x7f9af1af8675] /lib/libc.so.6(abort+0x17a)[0x7f9af1af991a] /usr/sbin/mysqld[0xa0fc7c] /usr/sbin/mysqld[0x981e0b] /usr/sbin/mysqld(_ZN22Sql_cmd_truncate_table14truncate_tableEP3THDP10TABLE_LIST+0x3a9)[0x84f209] /usr/sbin/mysqld(_ZN22Sql_cmd_truncate_table7executeEP3THD+0x5e)[0x84f2fe] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x2bb1)[0x6be431] /usr/sbin/mysqld(_Z11mysql_parseP3THDPcjP12Parser_state+0x398)[0x6c2188] /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcj+0x20a5)[0x6c42c5] /usr/sbin/mysqld(_Z24do_handle_one_connectionP3THD+0x1ed)[0x68bfcd] /usr/sbin/mysqld(handle_one_connection+0x42)[0x68c022] /usr/sbin/mysqld(pfs_spawn_thread+0x127)[0x90a597] /lib/libpthread.so.0(+0x7d56)[0x7f9af246fd56] /lib/libc.so.6(clone+0x6d)[0x7f9af1babefd] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (7f9984057070): is an invalid pointer Connection ID (thread ID): 273 Status: NOT_KILLED
[18 Sep 2014 7:16]
Vivek Padmanabhan
We have faced the same issue with Mysql Server version: 5.6.17-log; (We do drop and create tmp tables every 1 min from 40-60 different threads.) Attaching the detailed logs; 2014-09-18 06:25:58 4cc1d940 InnoDB: Assertion failure in thread 1287772480 in file row0mysql.cc line 3358 InnoDB: Failing assertion: !DICT_TF2_FLAG_IS_SET(table, DICT_TF2_TEMPORARY) InnoDB: We intentionally generate a memory trap. InnoDB: Submit a detailed bug report to http://bugs.mysql.com. InnoDB: If you get repeated assertion failures or crashes, even InnoDB: immediately after the mysqld startup, there may be InnoDB: corruption in the InnoDB tablespace. Please refer to InnoDB: http://dev.mysql.com/doc/refman/5.6/en/forcing-innodb-recovery.html InnoDB: about forcing recovery. 06:25:58 UTC - mysqld got signal 6 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. key_buffer_size=6442450944 read_buffer_size=1048576 max_used_connections=73 max_threads=151 thread_count=62 connection_count=62 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 6602773 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x2aaf8c102e50 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 4cc1d0a0 thread_stack 0x40000 /usr/local/mysql/bin/mysqld(my_print_stacktrace+0x35)[0x904cb5] /usr/local/mysql/bin/mysqld(handle_fatal_signal+0x3e8)[0x6718c8] /lib64/libpthread.so.0[0x3c8d40eb10] /lib64/libc.so.6(gsignal+0x35)[0x3c8cc30265] /lib64/libc.so.6(abort+0x110)[0x3c8cc31d10] /usr/local/mysql/bin/mysqld[0xa1f967] /usr/local/mysql/bin/mysqld[0x99c18a] /usr/local/mysql/bin/mysqld(_ZN22Sql_cmd_truncate_table16handler_truncateEP3THDP10TABLE_LISTb+0xab)[0x8846fb] /usr/local/mysql/bin/mysqld(_ZN22Sql_cmd_truncate_table14truncate_tableEP3THDP10TABLE_LIST+0x2f9)[0x884e49] /usr/local/mysql/bin/mysqld(_ZN22Sql_cmd_truncate_table7executeEP3THD+0x5e)[0x884ede] /usr/local/mysql/bin/mysqld(_Z21mysql_execute_commandP3THD+0x35e5)[0x6f2f05] /usr/local/mysql/bin/mysqld(_Z11mysql_parseP3THDPcjP12Parser_state+0x318)[0x6f6898] /usr/local/mysql/bin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcj+0xbcb)[0x6f7c6b] /usr/local/mysql/bin/mysqld(_Z10do_commandP3THD+0xd7)[0x6f9a67] /usr/local/mysql/bin/mysqld(_Z24do_handle_one_connectionP3THD+0x116)[0x6c1b26] /usr/local/mysql/bin/mysqld(handle_one_connection+0x45)[0x6c1c05] /usr/local/mysql/bin/mysqld(pfs_spawn_thread+0x126)[0x97e406] /lib64/libpthread.so.0[0x3c8d40673d] /lib64/libc.so.6(clone+0x6d)[0x3c8ccd3d1d] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (327a7d70): truncate xxxx_temp.xxxx Connection ID (thread ID): 195554 Status: NOT_KILLED The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. 140918 06:25:59 mysqld_safe Number of processes running now: 0 140918 06:25:59 mysqld_safe mysqld restarted 2014-09-18 06:26:00 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestamp server option (see documentation for more details). 2014-09-18 06:26:00 30690 [Note] Plugin 'FEDERATED' is disabled. 2014-09-18 06:26:00 2b0369d6d520 InnoDB: Warning: Using innodb_additional_mem_pool_size is DEPRECATED. This option may be removed in future releases, together with the option innodb_use_sys_malloc and with the InnoDB's internal memory allocator. 2014-09-18 06:26:00 30690 [Note] InnoDB: Using atomics to ref count buffer pool pages 2014-09-18 06:26:00 30690 [Note] InnoDB: The InnoDB memory heap is disabled 2014-09-18 06:26:00 30690 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2014-09-18 06:26:00 30690 [Note] InnoDB: Compressed tables use zlib 1.2.3 2014-09-18 06:26:00 30690 [Note] InnoDB: Using Linux native AIO 2014-09-18 06:26:00 30690 [Note] InnoDB: Using CPU crc32 instructions 2014-09-18 06:26:00 30690 [Note] InnoDB: Initializing buffer pool, size = 12.0G 2014-09-18 06:26:01 30690 [Note] InnoDB: Completed initialization of buffer pool 2014-09-18 06:26:01 30690 [Note] InnoDB: Highest supported file format is Barracuda. 2014-09-18 06:26:01 30690 [Note] InnoDB: Log scan progressed past the checkpoint lsn 17515376685696 2014-09-18 06:26:01 30690 [Note] InnoDB: Database was not shutdown normally! 2014-09-18 06:26:01 30690 [Note] InnoDB: Starting crash recovery. 2014-09-18 06:26:01 30690 [Note] InnoDB: Reading tablespace information from the .ibd files... 2014-09-18 06:26:01 30690 [ERROR] InnoDB: Attempted to open a previously opened tablespace. Previous tablespace mydb/mysqtable uses space ID: 3 at filepath: ./mydb/mysqtable.ibd. Cannot open tablespace mysql/slave_relay_log_info which uses space ID: 3 at filepath: ./mysql/slave_relay_log_info.ibd 2014-09-18 06:26:01 2b0369d6d520 InnoDB: Operating system error number 2 in a file operation. InnoDB: The error means the system cannot find the path specified. InnoDB: If you are installing InnoDB, remember that you must create InnoDB: directories yourself, InnoDB does not create them. InnoDB: Error: could not open single-table tablespace file ./mysql/slave_relay_log_info.ibd InnoDB: We do not continue the crash recovery, because the table may become InnoDB: corrupt if we cannot apply the log records in the InnoDB log to it. InnoDB: To fix the problem and start mysqld: InnoDB: 1) If there is a permission problem in the file and mysqld cannot InnoDB: open the file, you should modify the permissions. InnoDB: 2) If the table is not needed, or you can restore it from a backup, InnoDB: then you can remove the .ibd file, and InnoDB will do a normal InnoDB: crash recovery and ignore that table. InnoDB: 3) If the file system or the disk is broken, and you cannot remove InnoDB: the .ibd file, you can set innodb_force_recovery > 0 in my.cnf InnoDB: and force InnoDB to continue crash recovery here. 140918 06:26:02 mysqld_safe mysqld from pid file /data/mysqldata//machine.pid ended
[10 Oct 2014 4:55]
Jai Gupta
Me too - https://bugs.launchpad.net/percona-xtradb-cluster/+bug/1372263
[5 Dec 2014 7:45]
MySQL Verification Team
Going to look into this bug again shortly. Resolved stack trace from on reporter: [z@y tmp]$ addr2line --addresses --inlines --pretty-print --functions --demangle --exe=./mysqld 0xa1f967 0x99c18a 0x8846fb 0x884e49 0x884ede 0x6f2f05 0x6f6898 0x6f7c6b 0x6f9a67 0x6c1b26 0x6c1c05 0x97e406 0x0000000000a1f967: row_truncate_table_for_mysql(dict_table_t*, trx_t*) at ./mysql-5.6.17/storage/innobase/row/row0mysql.cc:3393 0x000000000099c18a: ha_innobase::truncate() at ./mysql-5.6.17/storage/innobase/handler/ha_innodb.cc:9921 0x00000000008846fb: Sql_cmd_truncate_table::handler_truncate(THD*, TABLE_LIST*, bool) at ./mysql-5.6.17/sql/sql_truncate.cc:234 0x0000000000884e49: Sql_cmd_truncate_table::truncate_table(THD*, TABLE_LIST*) at ./mysql-5.6.17/sql/sql_truncate.cc:429 0x0000000000884ede: Sql_cmd_truncate_table::execute(THD*) at ./mysql-5.6.17/sql/sql_truncate.cc:518 0x00000000006f2f05: mysql_execute_command(THD*) at ./mysql-5.6.17/sql/sql_parse.cc:4936 0x00000000006f6898: mysql_parse(THD*, char*, unsigned int, Parser_state*) at ./mysql-5.6.17/sql/sql_parse.cc:6236 0x00000000006f7c6b: dispatch_command(enum_server_command, THD*, char*, unsigned int) at ./mysql-5.6.17/sql/sql_parse.cc:1334 0x00000000006f9a67: do_command(THD*) at ./mysql-5.6.17/sql/sql_parse.cc:1036 0x00000000006c1b26: do_handle_one_connection(THD*) at ./mysql-5.6.17/sql/sql_connect.cc:982 0x00000000006c1c05: handle_one_connection at ./mysql-5.6.17/sql/sql_connect.cc:900 0x000000000097e406: pfs_spawn_thread at ./mysql-5.6.17/storage/perfschema/pfs.cc:1863
[2 Jan 2015 8:00]
MySQL Verification Team
I've never been able to repeat this.. Any ideas welcome!
[6 Feb 2015 15:51]
P McCormick
Right after upgrading to Percona XtraDB MySQL 5.6 on Ubuntu 14 (x86_64) I began having this crash every evening like clockwork. The crash occurred while xtrabackup was running, and was triggered by a sproc that had the following in it: CREATE TEMPORARY TABLE if not exists tmpTable ( A varchar(1000), B float, C varchar(1000)) ENGINE=MEMORY; TRUNCATE tmpInfusionTable; I changed the code so that instead of using truncate, I used drop then create. Since then, no more crashing. I don't have a development machine (or time) to try to reproduce this with a debug build, but it's a real bug and was quite repeatable on my production machine.
[11 Feb 2015 14:05]
MySQL Verification Team
I have repeated it once now on 5.6.22! InnoDB: Assertion failure in thread 24148 in file row0mysql.cc line 3418 InnoDB: Failing assertion: !DICT_TF2_FLAG_IS_SET(table, DICT_TF2_TEMPORARY) mysqld.exe!my_sigabrt_handler()[my_thr_init.c:458] mysqld.exe!raise()[winsig.c:593] mysqld.exe!abort()[abort.c:81] mysqld.exe!row_truncate_table_for_mysql()[row0mysql.cc:3420] mysqld.exe!ha_innobase::truncate()[ha_innodb.cc:9933] mysqld.exe!Sql_cmd_truncate_table::handler_truncate()[sql_truncate.cc:242] mysqld.exe!Sql_cmd_truncate_table::truncate_table()[sql_truncate.cc:450] mysqld.exe!Sql_cmd_truncate_table::execute()[sql_truncate.cc:543] mysqld.exe!mysql_execute_command()[sql_parse.cc:4946] mysqld.exe!mysql_parse()[sql_parse.cc:6364] mysqld.exe!dispatch_command()[sql_parse.cc:1335] mysqld.exe!do_command()[sql_parse.cc:1040] mysqld.exe!do_handle_one_connection()[sql_connect.cc:982] mysqld.exe!handle_one_connection()[sql_connect.cc:900] mysqld.exe!pfs_spawn_thread()[pfs.cc:1863] mysqld.exe!pthread_start()[my_winthread.c:63] mysqld.exe!_callthreadstartex()[threadex.c:314] mysqld.exe!_threadstartex()[threadex.c:292]
[12 Feb 2015 5:56]
MySQL Verification Team
repeated on linux, 5.6.24. It took couple of hours.
Attachment: bug72080_5.6.24_linux_crash.txt (text/plain), 138.81 KiB.
[12 Feb 2015 7:29]
MySQL Verification Team
Thanks to all the reporters here. Finally marking as verified as my testcase is somewhat repeatable :) -bash-4.1$ cat ./mysql-advanced-5.6.23-linux-glibc2.5-x86_64/data/x.err |grep -i "signal 6" 06:54:22 UTC - mysqld got signal 6 ; 07:09:50 UTC - mysqld got signal 6 ; 07:19:16 UTC - mysqld got signal 6 ; -bash-4.1$ cat ./mysql-advanced-5.6.23-linux-glibc2.5-x86_64/data/x.err |grep DICT InnoDB: Failing assertion: !DICT_TF2_FLAG_IS_SET(table, DICT_TF2_TEMPORARY) InnoDB: Failing assertion: !DICT_TF2_FLAG_IS_SET(table, DICT_TF2_TEMPORARY) InnoDB: Failing assertion: !DICT_TF2_FLAG_IS_SET(table, DICT_TF2_TEMPORARY)
[23 Feb 2015 14:35]
Daniel Price
Posted by developer: Fixed as of the upcoming 5.6.24, 5.7.7, 5.8.0 release, and here's the changelog entry: A "TRUNCATE TABLE" operation on a temporary table raised an assertion. The temporary table object was incompletely constructed when reloaded from "SYS_TABLES". Thank you for the bug report.
[27 Apr 2015 14:15]
Laurynas Biveinis
commit eefc5dd16f1046b836bc29346da23bba47470d74
Author: Thirunarayanan Balathandayuthapani <thirunarayanan.balathandayuth@oracle.com>
Date: Fri Feb 20 11:10:35 2015 +0530
Bug #20527363 TRUNCATE TEMPORARY TABLE CRASH:
!DICT_TF2_FLAG_IS_SET(TABLE, DICT_TF2_TEMPORARY)
Problem:
Server asserts while truncating the temporary tables.
Reason:
If temporary table object is evicted from data dictionary cache during its
lifetime on reload from SYS_TABLES object is not full constructed as it was
when it was created. This is because not all fields of temporary tables are
persisted to SYS_TABLES and dir_path_to_temp_table is one such field which
on reload is set to NULL. Making any decision based on such fields can result
in inconsistent output. Truncate code was trying to do the same.
Reviewed-by: Krunal Bauskar <krunal.bauskar@oracle.com>
Reviewed-by: Sunny Bains <sunny.bains@oracle.com>
RB: 8081
