Bug #8257 Record-file crash or full mysql crash while joining with a fulltext table
Submitted: 2 Feb 2005 9:13 Modified: 5 Feb 2005 16:49
Reporter: André Bieleman Email Updates:
Status: Can't repeat Impact on me:
None 
Category:MySQL Server: MyISAM storage engine Severity:S1 (Critical)
Version:4.1.8-Max OS:Linux (Fedora Core 1 & Redhat 7.2)
Assigned to: CPU Architecture:Any

[2 Feb 2005 9:13] André Bieleman
Description:
This is a problem which occured almost 1000 times in the last three weeks on one of our production servers running Redhat 7.2. Debugging is done on our development server running Fedora Core 1.

When performing a select with a join between two tables, the first one (k) using a range selection on a normal index and the second one (nf) using a fulltext index, we can get a error 127 (Record-file is crashed), error 134 (Record was already deleted (or record file crashed)) or the mysql server itself crashed.

The following table structures are appropriate:

CREATE TABLE `k` (
  `id` mediumint(8) unsigned NOT NULL default '0',
  `keyword` varchar(255) NOT NULL default '',
  PRIMARY KEY  (`keyword`,`id`)
) ENGINE=MyISAM DEFAULT CHARSET=latin1;

CREATE TABLE `nf` (
  `id` mediumint(8) unsigned NOT NULL default '0',
  `name` varchar(255) default NULL,
  `alias` varchar(255) default NULL,
  UNIQUE KEY `id` (`id`),
  FULLTEXT KEY `name` (`name`,`alias`)
) ENGINE=MyISAM DEFAULT CHARSET=latin1;

And the following queries gave an error:

mysql> SELECT nf.id FROM k, nf WHERE keyword IN('aa','bb') AND nf.id = k.id AND MATCH (nf.name,nf.alias) AGAINST ('+aa' IN BOOLEAN MODE);
ERROR 1030 (HY000): Got error 127 from storage engine

mysql> SELECT nf.id FROM k, nf WHERE keyword IN('aa','bb') AND nf.id = k.id AND MATCH (nf.name,nf.alias) AGAINST ('+aa +bb' IN BOOLEAN MODE);
ERROR 1030 (HY000): Got error 134 from storage engine

mysql> SELECT nf.id FROM k, nf WHERE keyword IN('aa','bb') AND nf.id = k.id AND MATCH (nf.name,nf.alias) AGAINST ('+aa +bb' IN BOOLEAN MODE);
ERROR 2013 (HY000): Lost connection to MySQL server during query

Here a part of the error log where the server crash exists:

mysqld got signal 11;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help diagnose
the problem, but since we have already crashed, something is definitely wrong
and this may fail.

key_buffer_size=8388608
read_buffer_size=131072
max_used_connections=3
max_connections=100
threads_connected=1
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_connections = 225791 K
bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

thd=0x895f010
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
Cannot determine thread, fp=0x47432cbc, backtrace may not be correct.
Stack range sanity check OK, backtrace follows:
0x8113e57
0xce20b8
0x8185d40
0x8185d40
0x81511e5
0x81500f8
0x8150179
0x814fe44
0x814678a
0x8146c52
0x8143de6
0x81238da
0x8127fa9
0x8122588
0x8121f54
0x8121607
0xcdb7fc
0xbf3aba
New value of fp=(nil) failed sanity check, terminating stack trace!
Please read http://dev.mysql.com/doc/mysql/en/Using_stack_trace.html and follow instructions on how to resolve the stack trace. Resolved
stack trace is much more helpful in diagnosing the problem, so please do
resolve it
Trying to get some variables.
Some pointers may be invalid and cause the dump to abort...
thd->query at 0x898ad80 = SELECT nf.id FROM k, nf WHERE keyword IN('aa','bb') AND nf.id = k.id AND MATCH (nf.name,nf.alias) AGAINST ('+aa +bb' IN BOOLEAN MODE)
thd->thread_id=12
The manual page at http://www.mysql.com/doc/en/Crashing.html contains
information that should help you find out what is causing the crash.

And:

050201 15:38:24 [ERROR] Got error 134 when reading table './debug/nf'
050201 16:47:23 [ERROR] Got error 127 when reading table './debug/nf'
050201 16:47:26 [ERROR] Got error 127 when reading table './debug/nf'
050202  9:22:29 [ERROR] Got error 127 when reading table './debug/nf'
050202  9:23:36 [ERROR] Got error 127 when reading table './debug/nf'
050202  9:23:40 [ERROR] Got error 127 when reading table './debug/nf'

Currently I'am trying to repeat the errors on a much smaller data set. Table k holds over 10 million records and table nf over 1 million. When using only the structures with a few records dummy data the errors didn't occur.

How to repeat:
Use table structures and queries listed above. But... not yet repeatable without a bunch of records!

Suggested fix:
None.
[2 Feb 2005 9:25] André Bieleman
Forgot to explicit mention that when not using the boolean mode or when no range lookup is performed the query goes well:

mysql> SELECT nf.id FROM k, nf WHERE keyword IN('aa','bb') AND nf.id = k.id AND MATCH (nf.name,nf.alias) AGAINST ('+aa +bb' IN BOOLEAN MODE);
ERROR 2013 (HY000): Lost connection to MySQL server during query
mysql> SELECT nf.id FROM k, nf WHERE keyword IN('aa') AND nf.id = k.id AND MATCH (nf.name,nf.alias) AGAINST ('+aa +bb' IN BOOLEAN MODE);
ERROR 2006 (HY000): MySQL server has gone away
No connection. Trying to reconnect...
Connection id:    11
Current database: debug

Empty set (0.01 sec)

mysql> SELECT nf.id FROM k, nf WHERE keyword IN('aa','bb') AND nf.id = k.id AND MATCH (nf.name,nf.alias) AGAINST ('aa bb');   +---------+
| id      |
+---------+
|  313762 |
|  464294 |
|  496765 |
|  565624 |
| 1024669 |
|  178280 |
|  979481 |
+---------+
7 rows in set (1.38 sec)

We also have ft_min_word_len = 2 and a custom stopword list.
[5 Feb 2005 16:49] Jorge del Conde
Hi!

I was unable to reproduce this behaviour.  By the error messages that you received, it looks like your tables have become corrupted, and unless you provide us with a test-case that reproduces this behaviour, there isn't much we can do.  Can you please give us a test-case that re-creates this scenario ?

Thanks !
[21 Feb 2005 9:04] André Bieleman
Tables where certainly not corrupted, because I made new tables like the old ones (CREATE TABLE LIKE) and inserted the data with a INSERT INTO ... SELECT.

A random filled table k with 4,4 million records joined with the nf table did also crash the server. I still can't repeat with a random filled fulltext table (nf).

But, I'm very pleased to see that the problem is fixed in 4.1.10!! Looks like fix: 'Fixed a crash in a boolean full-text search in certain joins. (Bug #8234)' did it.