Bug #67960 innodb_ft_user_stopword_table becomes corrupt, causes server crash
Submitted: 21 Dec 2012 17:08 Modified: 12 Apr 2013 12:28
Reporter: Kolbe Kegel Email Updates:
Status: Closed Impact on me:
None 
Category:MySQL Server: InnoDB storage engine Severity:S1 (Critical)
Version:5.6.9 OS:Any
Assigned to: Jimmy Yang CPU Architecture:Any
Tags: crash, fulltext, innodb

[21 Dec 2012 17:08] Kolbe Kegel
Description:
In some situations, the innodb_ft_user_stopword_table variable can become corrupted, and the MySQL server can crash.

Repeating this is not difficult, but I also haven't yet found it to be deterministic.

How to repeat:
DROP TABLE IF EXISTS i1, i1_stopwords, no_stopwords;
CREATE TABLE `i1_stopwords` (
  `value` varchar(255) DEFAULT NULL
) ENGINE=InnoDB;

INSERT INTO `i1_stopwords` VALUES ('apple'),('banana'),('coconut'),('durian');

CREATE TABLE `no_stopwords` (
  `value` varchar(32) DEFAULT NULL
) ENGINE=InnoDB;

set global innodb_ft_user_stopword_table='test/i1_stopwords';
set session innodb_ft_user_stopword_table='test/i1_stopwords';

CREATE TABLE `i1` (
  `id` int(10) unsigned NOT NULL AUTO_INCREMENT,
  `c` varchar(32) DEFAULT NULL,
  PRIMARY KEY (`id`),
  FULLTEXT KEY `c` (`c`)
) ENGINE=InnoDB;

INSERT INTO `i1` VALUES (1,'a crabby apple'),(2,'the banana bonanza'),(3,'some crazy coconuts'),(4,'one daring durian');

set innodb_ft_user_stopword_table='test/no_stopwords';
select @@innodb_ft_user_stopword_table;

optimize table i1;

select @@innodb_ft_user_stopword_table;

set session innodb_ft_user_stopword_table='test/no_stopwords';

optimize table i1;

connect;

set global innodb_ft_user_stopword_table='test/no_stopwords';

optimize table i1;

select @@innodb_ft_user_stopword_table;

#select sleep(5);

set session innodb_ft_user_stopword_table='test/no_stopwords';

set global innodb_ft_user_stopword_table='test/no_stopwords';

set session innodb_ft_user_stopword_table='test/no_stopwords';

set global innodb_ft_user_stopword_table='test/no_stopwords';

set session innodb_ft_user_stopword_table='test/no_stopwords';

set global innodb_ft_user_stopword_table='test/no_stopwords';
[21 Dec 2012 17:15] Kolbe Kegel
Here are some examples of what I have seen:

=================================================
mysql 5.6.9-rc (root) [test]> set innodb_ft_user_stopword_table='test/no_stopwords';
ERROR 2013 (HY000): Lost connection to MySQL server during query
=================================================

=================================================
mysql 5.6.9-rc (root) [test]> select @@innodb_ft_user_stopword_table;                                                                                         ERROR 2006 (HY000): MySQL server has gone away
=================================================

=================================================
mysql 5.6.9-rc (root) [test]> show variables like 'innodb_ft_user_stopword_table';
+-------------------------------+-------+
| Variable_name                 | Value |
+-------------------------------+-------+
| innodb_ft_user_stopword_table |       |
+-------------------------------+-------+
1 row in set, 1 warning (0.00 sec)

mysql 5.6.9-rc (root) [test]> show warnings;
+---------+------+--------------------------------------------------------------------------------------+
| Level   | Code | Message                                                                              |
+---------+------+--------------------------------------------------------------------------------------+
| Warning | 1366 | Incorrect string value: '\x98\xAC\x99\x1A\xE3' for column 'VARIABLE_VALUE' at row 1 |
+---------+------+--------------------------------------------------------------------------------------+
1 row in set (0.00 sec)
=================================================

=================================================
mysql 5.6.9-rc (root) [test]> select @@global.innodb_ft_user_stopword_table;
+----------------------------------------+
| @@global.innodb_ft_user_stopword_table |
+----------------------------------------+
| O?o?                                   |
+----------------------------------------+
1 row in set (0.00 sec)

mysql 5.6.9-rc (root) [test]> show variables like 'innodb_ft_user_stopword_table';
+-------------------------------+-------+
| Variable_name                 | Value |
+-------------------------------+-------+
| innodb_ft_user_stopword_table | O    |
+-------------------------------+-------+
1 row in set, 1 warning (0.00 sec)

mysql 5.6.9-rc (root) [test]> show warnings;
+---------+------+------------------------------------------------------------------------------+
| Level   | Code | Message                                                                      |
+---------+------+------------------------------------------------------------------------------+
| Warning | 1366 | Incorrect string value: '\xB9o\xFE\x07' for column 'VARIABLE_VALUE' at row 1 |
+---------+------+------------------------------------------------------------------------------+
1 row in set (0.00 sec)
=================================================

=================================================
mysql 5.6.9-rc (root) [test]> select @@innodb_ft_user_stopword_table;
+---------------------------------+
| @@innodb_ft_user_stopword_table |
+---------------------------------+
| test/i1_stopwords               |
+---------------------------------+
1 row in set (0.01 sec)

mysql 5.6.9-rc (root) [test]> set innodb_ft_user_stopword_table='test/no_stopwords';                                                                          Query OK, 0 rows affected (0.00 sec)

mysql 5.6.9-rc (root) [test]> \r
Connection id:    4
Current database: test

mysql 5.6.9-rc (root) [test]> select @@innodb_ft_user_stopword_table;
+---------------------------------+
| @@innodb_ft_user_stopword_table |
+---------------------------------+
|                                 |
+---------------------------------+
1 row in set (0.00 sec)

mysql 5.6.9-rc (root) [test]> optimize table i1;
+---------+----------+----------+-------------------------------------------------------------------+
| Table   | Op       | Msg_type | Msg_text                                                          |
+---------+----------+----------+-------------------------------------------------------------------+
| test.i1 | optimize | note     | Table does not support optimize, doing recreate + analyze instead |
| test.i1 | optimize | status   | OK                                                                |
+---------+----------+----------+-------------------------------------------------------------------+
2 rows in set (0.07 sec)

mysql 5.6.9-rc (root) [test]> select @@innodb_ft_user_stopword_table;                                                                                         +---------------------------------+
| @@innodb_ft_user_stopword_table |
+---------------------------------+
| ?A0?                            |
+---------------------------------+
1 row in set (0.00 sec)
=================================================
[21 Dec 2012 17:16] Kolbe Kegel
bt full from 5.6.9 on Linux, SIGABRT during OPTIMIZE TABLE

Attachment: gdb.txt (text/plain), 24.47 KiB.

[21 Dec 2012 17:19] Kolbe Kegel
Error log backtrace from mysql-5.6.9-rc-osx10.7-x86_64 while trying to execute "set session innodb_ft_user_stopword_table='test/no_stopwords'":

mysqld(62116,0x117ee1000) malloc: *** error for object 0x7fdd62926570: pointer being freed was not allocated
*** set a breakpoint in malloc_error_break to debug
06:29:16 UTC - mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed, 
something is definitely wrong and this may fail.

key_buffer_size=8388608
read_buffer_size=131072
max_used_connections=2
max_threads=151
thread_count=1
connection_count=1
It is possible that mysqld could use up to 
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 68208 K  bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0x7fdd622aaa00
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 117ee0e88 thread_stack 0x40000
0   mysqld                              0x000000010839da1e my_print_stacktrace + 44^@
1   mysqld                              0x00000001081fed14 handle_fatal_signal + 725^@
2   libsystem_c.dylib                   0x00007fff8df678ea _sigtramp + 26^@
3   mysqld                              0x0000000108190ef0 _ZN15Item_func_sleep7val_intEv + 806^@
4   libsystem_c.dylib                   0x00007fff8dfbedce abort + 143^@
5   libsystem_c.dylib                   0x00007fff8df92959 free + 392^@
6   mysqld                              0x00000001082a3495 _ZN17sys_var_pluginvar14session_updateEP3THDP7set_var + 283^@
7   mysqld                              0x00000001081fd7c5 _ZN7sys_var6updateEP3THDP7set_var + 243^@
8   mysqld                              0x00000001081fe334 _ZN7set_var6updateEP3THD + 26^@
9   mysqld                              0x00000001081fe0a6 _Z17sql_set_variablesP3THDP4ListI12set_var_baseE + 88^@
10  mysqld                              0x0000000108286636 _Z21mysql_execute_commandP3THD + 10952^@
11  mysqld                              0x0000000108283327 _Z11mysql_parseP3THDPcjP12Parser_state + 601^@
12  mysqld                              0x00000001082815f0 _Z16dispatch_command19enum_server_commandP3THDPcj + 1902^@
13  mysqld                              0x0000000108282f01 _Z10do_commandP3THD + 370^@
14  mysqld                              0x00000001082488df _Z24do_handle_one_connectionP3THD + 280^@
15  mysqld                              0x00000001082487ba handle_one_connection + 59^@
16  mysqld                              0x000000010853cd92 pfs_spawn_thread + 354^@
17  libsystem_c.dylib                   0x00007fff8df79742 _pthread_start + 327^@
18  libsystem_c.dylib                   0x00007fff8df66181 thread_start + 13^@

Trying to get some variables.
Some pointers may be invalid and cause the dump to abort.
Query (7fdd622ba010): is an invalid pointer
Connection ID (thread ID): 2
Status: NOT_KILLED
[21 Dec 2012 18:13] Sinisa Milivojevic
Kolbe,

Do I understand you correctly. You just enter non-existent table in a variable and that crashes the server ???

If not, how do you corrupt the variable ????
[21 Dec 2012 18:14] Shane Bester
Well, I can repeat it :

>mysqld-debug.exe!_output_s_l
 mysqld-debug.exe!_vsnprintf_helper
 mysqld-debug.exe!_vsnprintf_s_l 
 mysqld-debug.exe!_vsnprintf_s 
 mysqld-debug.exe!_VCrtDbgReportA 
 mysqld-debug.exe!_CrtDbgReportV
 mysqld-debug.exe!_CrtDbgReport
 mysqld-debug.exe!_free_dbg_nolock
 mysqld-debug.exe!_free_dbg
 mysqld-debug.exe!free
 mysqld-debug.exe!my_free
 mysqld-debug.exe!innodb_session_stopword_update
 mysqld-debug.exe!sys_var_pluginvar::session_update
 mysqld-debug.exe!sys_var::update
 mysqld-debug.exe!set_var::update
 mysqld-debug.exe!sql_set_variables
 mysqld-debug.exe!mysql_execute_command
 mysqld-debug.exe!mysql_parse
 mysqld-debug.exe!dispatch_command
 mysqld-debug.exe!do_command
 mysqld-debug.exe!do_handle_one_connection
 mysqld-debug.exe!handle_one_connection
 mysqld-debug.exe!pfs_spawn_thread
 mysqld-debug.exe!pthread_start
 mysqld-debug.exe!_callthreadstartex
 mysqld-debug.exe!_threadstartex
 
 
But, I propose we write a proper fuzz tester to check this millions of times and catch all occurrences under valgrind.
[21 Dec 2012 18:17] Shane Bester
Sinisa, write a script to put in pseudo-random characters (not a dash!) into every place a ? appears below and run it in a loop :)

----
drop table if exists `t?`
create table if not exists `t?`(value varchar(20),fulltext index(value))engine=innodb
create table if not exists `t?`(value varchar(10))engine=innodb
select @@session.innodb_ft_user_stopword_table
select @@global.innodb_ft_user_stopword_table
set global innodb_ft_user_stopword_table='test/t?'
set session innodb_ft_user_stopword_table='test/t?'
insert into `t?` values('?')
optimize table `t?`
------
[21 Dec 2012 18:45] Kolbe Kegel
Nice work, Shane. Thanks!

I guess this should be Verified now, and you do not "Need Feedback" from me?

Kolbe
[22 Dec 2012 11:02] Shane Bester
sample testcase....

Attachment: bug67960.php (application/octet-stream, text), 1.37 KiB.

[6 Feb 2013 8:52] Erlend Dahl
This was fixed in 5.6.10.
[12 Apr 2013 12:28] Paul Dubois
Noted in 5.6.10 changelog.

Corruption of the innodb_ft_user_stopword_table table could cause a
server exit.