Bug #29464 load data infile into table with big5 chinese fulltext index hangs 100% cpu

Description:
completes in <5 seconds on 5.0.44.

5.1.21 and 6.0.1 seems to hang:

mysql> show processlist;
+----+------+------+------+---------+------+-------------------+------------------------
| Id | User | Host | db   | Command | Time | State             | Info
+----+------+------+------+---------+------+-------------------+------------------------
|  3 | root |      | test | Query   |  471 | Repair by sorting | load data local infile
|  5 | root |      | a    | Query   |    0 | NULL              | show processlist
+----+------+------+------+---------+------+-------------------+------------------------

 PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND                                                                                                              
 4712 sbester   16   0 69336  34m 5392 S 99.3  1.7  10:50.90 mysqld       

How to repeat:
drop table if exists tbl_0;
create table tbl_0 (a varchar(2) character set big5 collate big5_chinese_ci,b varchar(2) character set big5 collate big5_chinese_ci,fulltext key(a,b),key(b,a),key(b),key(a))engine=myisam;
load data local infile '/tmp/dump.txt' replace into table tbl_0 (@h,@i) set a=UNHEX(@h),b=UNHEX(@h);
check table tbl_0 extended;

Suggested fix:
.

dump.txt used in the testcase.

Attachment: dump.txt (text/plain), 105.25 KiB.

Maybe irrelevant but I caught it in debugger. In ft_simple_get_word() there's "uint mwc, length, mbl;" but somewhere mbl is getting a return value of -2 which turns into 4294967294, hence a >0 check presumably misbehaves.

A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/30365

ChangeSet@1.2528, 2007-07-05 15:17:06+05:00, svoj@mysql.com +3 -0
  BUG#29464 - load data infile into table with big5 chinese fulltext index
              hangs 100% cpu
  
  Fulltext parser may fall into infinite loop when it gets illegal
  multibyte sequence (or a sequence that doesn't have mapping to unicode).
  
  Affects 5.1 only.

A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/30828

ChangeSet@1.2545, 2007-07-13 03:29:25+05:00, svoj@mysql.com +4 -0
  BUG#29464 - load data infile into table with big5 chinese fulltext index
              hangs 100% cpu
  
  Moved a test case for BUG#29464 into fulltext3.test, since it requires big5
  character set.

Pushed into 5.1.21-beta

Noted in 5.1.21 changelog.

The full-text parser could enter an infinite loop if it encountered
an illegal multi-byte sequence or a sequence that has no mapping to
Unicode.

Pushed into 5.1.25-maria-alpha

Pushed into 5.1.25-maria-alpha

Pushed into 5.1.25-maria-alpha

Pushed into 5.1.25-maria-alpha

Pushed into 5.1.25-maria-alpha

Pushed into 5.1.25-maria-alpha

Pushed into 5.1.25-maria-alpha

Pushed into 5.1.25-maria-alpha

Pushed into 5.1.25-maria-alpha

Pushed into 5.1.25-maria-alpha

Pushed into 5.1.25-maria-alpha

Pushed into 5.1.25-maria-alpha

Pushed into 5.1.25-maria-alpha