Bug #72557 Prefixed wildcard search causes MySQL to crash (assertion error in fts0que.cc)
Submitted: 7 May 2014 7:45 Modified: 8 Jun 2014 12:30
Reporter: Benjamin Wilher Email Updates:
Status: No Feedback Impact on me:
None 
Category:MySQL Server: FULLTEXT search Severity:S2 (Serious)
Version:5.6.10 OS:Linux (linux-glibc2.5-x86_64)
Assigned to: CPU Architecture:Any

[7 May 2014 7:45] Benjamin Wilher
Description:
I got a fairly big table (~25,000,000 rows, 4.0GB, index size 1.3GB) with a FULLTEXT-indexed data column. The table gets updated several times a day. The indexed data column contains a compound string of several terms (text and numbers)
Since a couple of days the mysql server occationally crashes and restarts with the following error log output:

------------------ error.log ----------------------------------------------------------

2014-05-06 10:02:23 7f8a1d4a1700  InnoDB: Assertion failure in thread 140231173609216 in file fts0que.cc line 3142
InnoDB: Failing assertion: ret == 0
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: http://dev.mysql.com/doc/refman/5.6/en/forcing-innodb-recovery.html
InnoDB: about forcing recovery.
08:02:23 UTC - mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed, 
something is definitely wrong and this may fail.

key_buffer_size=402653184
read_buffer_size=2097152
max_used_connections=17
max_threads=1000
thread_count=13
connection_count=12
It is possible that mysqld could use up to 
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 2710731 K  bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0x32aacec0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 7f8a1d4a0e10 thread_stack 0x40000
/usr/local/mysql/bin/mysqld(my_print_stacktrace+0x35)[0x8f0d25]
/usr/local/mysql/bin/mysqld(handle_fatal_signal+0x3e8)[0x66b138]
/lib64/libpthread.so.0[0x373e20f710]
/lib64/libc.so.6(gsignal+0x35)[0x373da32925]
/lib64/libc.so.6(abort+0x175)[0x373da34105]
/usr/local/mysql/bin/mysqld[0xa79f23]
/usr/local/mysql/bin/mysqld[0x903ac9]
/usr/local/mysql/bin/mysqld(_ZN15Item_func_match11init_searchEb+0x15b)[0x5f6a3b]
/usr/local/mysql/bin/mysqld(_Z12init_ftfuncsP3THDP13st_select_lexb+0x5e)[0x6982fe]
/usr/local/mysql/bin/mysqld(_ZN4JOIN8optimizeEv+0x3ab9)[0x85bce9]
/usr/local/mysql/bin/mysqld[0x706474]
/usr/local/mysql/bin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_P10SQL_I_ListI8st_orderESB_S7_yP13select_resultP18st_select_lex_unitP13st_select_lex+0xbc)[0x7067fc]
/usr/local/mysql/bin/mysqld(_Z13handle_selectP3THDP13select_resultm+0x175)[0x706a05]
/usr/local/mysql/bin/mysqld[0x6e3ac9]
/usr/local/mysql/bin/mysqld(_Z21mysql_execute_commandP3THD+0x3ab9)[0x6e8899]
/usr/local/mysql/bin/mysqld(_Z11mysql_parseP3THDPcjP12Parser_state+0x318)[0x6eba58]
/usr/local/mysql/bin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcj+0x827)[0x6ec377]
/usr/local/mysql/bin/mysqld(_Z10do_commandP3THD+0xd7)[0x6ed617]
/usr/local/mysql/bin/mysqld(_Z24do_handle_one_connectionP3THD+0x116)[0x6b7e06]
/usr/local/mysql/bin/mysqld(handle_one_connection+0x45)[0x6b7ee5]
/usr/local/mysql/bin/mysqld(pfs_spawn_thread+0x13b)[0xab465b]
/lib64/libpthread.so.0[0x373e2079d1]
/lib64/libc.so.6(clone+0x6d)[0x373dae8b6d]

Trying to get some variables.
Some pointers may be invalid and cause the dump to abort.
Query (7f89e8004c30): is an invalid pointer
Connection ID (thread ID): 7
Status: NOT_KILLED

The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you find out what is causing the crash.
140506 10:02:23 mysqld_safe Number of processes running now: 0
140506 10:02:23 mysqld_safe mysqld restarted

----------------------------------------------------------------------------------------------

How to repeat:
We could identify the query which caused the problem:

SELECT entry_id FROM entrysearchindex WHERE MATCH(entry_content) AGAINST('+100*' IN BOOLEAN MODE)

The table structure used:

CREATE TABLE `entrysearchindex` (
	`entry_id` INT(11) UNSIGNED NOT NULL,
	`entry_lastsynced` TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
	`entry_content` TEXT NOT NULL,
	PRIMARY KEY (`entry_id`),
	INDEX `ix_lastsynced` (`entry_lastsynced`),
	FULLTEXT INDEX `ft_content` (`entry_content`),
	CONSTRAINT `fk_entry` FOREIGN KEY (`entry_id`) REFERENCES `entry` (`entry_id`) ON UPDATE CASCADE ON DELETE CASCADE
)
COLLATE='utf8_general_ci'
ENGINE=InnoDB;
[7 May 2014 10:22] Benjamin Wilher
Another crash was caused by this WHERE expression 'MATCH(entry_content) AGAINST('an' IN BOOLEAN MODE)'
[7 May 2014 14:45] MySQL Verification Team
Please check against 5.6.17 version. Thanks.
[8 May 2014 8:24] Benjamin Wilher
5.6.17 required me to rebuild the index, because of a bug which is not solved until 5.6.19.

But we've rebuilt the index anyway (using the same version 5.6.10). Now the server doesn't crash anymore. This proves, that the index was broken in some way. The strange thing is, that we have two exactly identical setups on two servers, which are not connected to each other. The first crashed on 3rd May, the second one crashed on the 6th. The index was created about a year ago and did not make any problem since now.
[8 May 2014 12:30] MySQL Verification Team
Thank you for the feedback. Please check for duplicate with http://bugs.mysql.com/bug.php?id=68987 where a FTS index is involved. Thanks.
[9 Jun 2014 1:00] Bugs System
No feedback was provided for this bug for over a month, so it is
being suspended automatically. If you are able to provide the
information that was originally requested, please do so and change
the status of the bug back to "Open".