MySQL Bugs: #19741: segfault with cp1250 charset + like + primary key + 64bit os

Bug #19741	segfault with cp1250 charset + like + primary key + 64bit os
Submitted:	11 May 2006 20:55	Modified:	12 Sep 2006 0:44
Reporter:	pavel stano	Email Updates:
Status:	Closed	Impact on me:	None
Category:	MySQL Server: Charsets	Severity:	S2 (Serious)
Version:	4.1.19	OS:	64bit linux
Assigned to:	Alexander Barkov	CPU Architecture:	Any

Description:
on 64bit OS, mysqld segfault in some situation when you use cp1250 charset + WHERE LIKE + primary key +  characters with diacritic marks

How to repeat:
// my terminal is iso-8859-2
mysql> set names latin2;
mysql> create table a (a varchar(15) collate cp1250_czech_cs NOT NULL, primary key(a));
mysql> insert into a values("abcdefghá");
mysql> insert into a values("ááčč");
mysql> SELECT a FROM a WHERE a LIKE "abcdefghá";
ERROR 2013 (HY000): Lost connection to MySQL server during query

Hi Pavel. Thanks for your bug report.
I was able to verify it exactly like you described.
My environment: 
  MySQL 4.1.20 bk build
  FC5x64
  AMDx64
  3GB RAM
  Terminal in iso-8859-2
Server started with no defaults
060514 21:29:08  mysqld started
060514 21:29:09  InnoDB: Started; log sequence number 0 43634
/home/tonci/mysql4-1/libexec/mysqld: ready for connections.
Version: '4.1.20-debug'  socket: '/tmp/mysql.sock'  port: 3306  Source distribution

mysql> use test;
mysql> set names latin2;
mysql> create table a (a varchar(15) collate cp1250_czech_cs NOT NULL, primary key(a));
mysql> insert into a values("abcdefghá");
mysql> insert into a values("ááčč");
mysql> SELECT a FROM a WHERE a LIKE "abcdefghá";
ERROR 2013 (HY000): Lost connection to MySQL server during query

Error log:
mysqld got signal 11;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help diagnose
the problem, but since we have already crashed, something is definitely wrong
and this may fail.

key_buffer_size=8388572
read_buffer_size=131072
max_used_connections=1
max_connections=100
threads_connected=1
It is possible that mysqld could use up to 
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_connections = 225788 K
bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

Number of processes running now: 0
060514 21:48:47  mysqld restarted
060514 21:48:48  InnoDB: Started; log sequence number 0 43634
/home/tonci/mysql4-1/libexec/mysqld: ready for connections.
Version: '4.1.20-debug'  socket: '/tmp/mysql.sock'  port: 3306  Source distribution

I've tested without "set names...", same crash happens.

i debug this bug on mysql 5.0.22 and i think i found the problem, here is backtrace
#0  0x0000366e3b905c83 in pthread_kill () from /lib64/tls/libpthread.so.0
#1  0x0000000000521b1a in handle_segfault (sig=11) at mysqld.cc:2094
#2  <signal handler called>
#3  0x00000000006bb15a in my_like_range_win1250ch (cs=0xa05e00, ptr=0x4007d848 "á", ptr_length=1, 
    escape=92 '\\', w_one=95 '_', w_many=37 '%', res_length=15, min_str=0xa91578 "Ť", 
    max_str=0xa91587 "ions.MYI", min_length=0x4007d7f8, max_length=0xffffffe1)
    at ctype-win1250ch.c:639
#4  0x00000000005ca616 in get_mm_leaf (param=0x4007ddd0, conf_func=0x9, field=0xa658c0, 
    key_part=0xa9157f, type=BETWEEN, value=0x4007d840) at sql_string.h:90
#5  0x00000000005ca19c in get_mm_parts (param=0x4007ddd0, cond_func=0xa8f428, field=0xa658c0, 
    type=LIKE_FUNC, value=0xa8f330, cmp_type=37) at opt_range.cc:3838
#6  0x00000000005c977b in get_func_mm_tree (param=0x4007ddd0, cond_func=0xa8f428, field=0xa658c0, 
    value=0xa8f330, cmp_type=STRING_RESULT, inv=false) at opt_range.cc:3623
#7  0x00000000005c9eb6 in get_mm_tree (param=0x4007ddd0, cond=0xa658c0) at opt_range.cc:3792
#8  0x00000000005c6522 in SQL_SELECT::test_quick_select (this=0xac9aa8, thd=0xa73d30, keys_to_use=
      {map = 4609150658637164514}, prev_tables=0, limit=0, force_quick_range=false)
    at opt_range.cc:1848

if you look at function my_like_range in libmysql/ctype-win1250ch.c on line 639
*min_str = like_range_prefix_min_win1250ch[(uint)(*ptr)];
when mysqld crash, *ptr was "á"
like_range_prefix_min_win1250ch is an array of chars
the problem is that (uint)(*ptr)=4294967265
it is because *ptr is signed char and "á"=-31 converted to unsigned int = 4294967265
when you change *ptr to unsigned char then it dont crash, because unsigned char *ptr="á"==225
but i dont know why it dont crash on 32bit OS on vanila mysql

there is a possibility that i am wrong :)

A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/9534

Merged into 5.1.12

Noted in 4.1.22, 5.0.25, 5.1.12 changelogs.

On 64-bit systems, use of the cp1250 character set with a primary key
column in a LIKE clause caused a server crash for patterns having
letters in the range 128..255.