Bug #38701 Crash in String::append when inserting duplicate empty strings an uft8 SET col
Submitted: 10 Aug 2008 17:50 Modified: 17 Oct 2008 18:23
Reporter: Philip Stoev Email Updates:
Status: Closed Impact on me:
None 
Category:MySQL Server Severity:S1 (Critical)
Version:5.1.28,6.0.7 OS:Any
Assigned to: Georgi Kodinov CPU Architecture:Any
Tags: regression

[10 Aug 2008 17:50] Philip Stoev
Description:
When executing a query that would insert a duplicate empty string in an uft8 SET column with at least 20 different possible values, mysqld crashes as follows:

#5  0x00432f5f in memcpy () from /lib/libc.so.6
#6  0x082ece26 in String::append (this=0xad27d0cc, s=@0xad27c070) at sql_string.cc:351
#7  0x082d5fbd in key_unpack (to=0xad27d0cc, table=0x9dc9248, idx=0) at key.cc:391
#8  0x0841ae07 in handler::print_keydup_error (this=0x9dc9920, key_nr=0, msg=0x9d269f3 "Duplicate entry '%-.64s' for key '%-.192s'") at handler.cc:2530
#9  0x0841afc0 in handler::print_error (this=0x9dc9920, error=121, errflag=0) at handler.cc:2583
#10 0x08391906 in write_record (thd=0x9d75d90, table=0x9dc9248, info=0xad27e36c) at sql_insert.cc:1603
#11 0x08395913 in mysql_insert (thd=0x9d75d90, table_list=0x9dd1a20, fields=@0x9d77564, values_list=@0x9d77588, update_fields=@0x9d7757c,
    update_values=@0x9d77570, duplic=DUP_ERROR, ignore=false) at sql_insert.cc:828
#12 0x08307ac2 in mysql_execute_command (thd=0x9d75d90) at sql_parse.cc:2944
#13 0x0830e1e1 in mysql_parse (thd=0x9d75d90, inBuf=0x9dd1720 "INSERT INTO C ( set_unique_utf8 ) VALUES ( '' )", length=47, found_semicolon=0xad27f270)
    at sql_parse.cc:5800
#14 0x0830ec55 in dispatch_command (command=COM_QUERY, thd=0x9d75d90, packet=0x9dc2b99 "INSERT INTO C ( set_unique_utf8 ) VALUES ( '' )", packet_length=47)
    at sql_parse.cc:1050
#15 0x0830ff1f in do_command (thd=0x9d75d90) at sql_parse.cc:723
#16 0x082fc99b in handle_one_connection (arg=0x9d75d90) at sql_connect.cc:1153
#17 0x0057d32f in start_thread () from /lib/libpthread.so.0
#18 0x0049a27e in clone () from /lib/libc.so.6

This is the problematic memcpy:

346     {
347       if (s.length())
348       {
349         if (realloc(str_length+s.length()))
350           return TRUE;
351         memcpy(Ptr+str_length,s.ptr(),s.length());
352         str_length+=s.length();
353       }
354       return FALSE;
355     }

How to repeat:
CREATE TABLE C (
        set_unique_utf8 set ('a','b','c','d','e','f','g','h','i','j','k','l','m','n','o','p','q','r','s','t','u','v','w','x','y','z')
        CHARACTER SET utf8,
        unique (set_unique_utf8)
);
INSERT INTO C ( set_unique_utf8 ) VALUES ( '' );
INSERT INTO C ( set_unique_utf8 ) VALUES ( '' );

Suggested fix:
It appears that this is one of those bugs where a buffer that was sufficient to hold enough data using latin1 is no longer sufficient with utf8.
[11 Aug 2008 3:43] Valeriy Kravchuk
Thank you for a bug report. Verified just as described with 6.0.5 on Windows XP. I've got the following stack trace:

mysqld.exe!String::append
mysqld.exe!key_unpack
mysqld.exe!handler::print_keydup_error
mysqld.exe!handler::print_error
mysqld.exe!write_record
mysqld.exe!mysql_insert
mysqld.exe!mysql_execute_command
mysqld.exe!mysql_parse
mysqld.exe!dispatch_command
mysqld.exe!do_command
mysqld.exe!handle_one_connection
mysqld.exe!pthread_start
mysqld.exe!_threadstart
...
[11 Aug 2008 16:12] MySQL Verification Team
bug is in 5.1.28 also:
 Conditional jump or move depends on uninitialised value(s)
: handler::print_keydup_error  (sql_string.h:102)
: handler::print_error  (handler.cc:2542)
: write_record  (sql_insert.cc:1585)
: mysql_insert  (sql_insert.cc:812)
: mysql_execute_command (sql_parse.cc:2955)
: mysql_parse  (sql_parse.cc:5656)
: dispatch_command  (sql_parse.cc:1137)
: do_command  (sql_parse.cc:794)
: handle_one_connection (sql_connect.cc:1115)
[11 Aug 2008 16:39] Valeriy Kravchuk
Does not crash 4.1.22:

C:\Program Files\MySQL\MySQL Server 5.1\bin>mysql -uroot -proot -P3306 test
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 1
Server version: 4.1.22-community-nt-log

Type 'help;' or '\h' for help. Type '\c' to clear the buffer.

mysql> CREATE TABLE C (
    ->         set_unique_utf8 set
    -> ('a','b','c','d','e','f','g','h','i','j','k','l','m','n','o','p','q','r',
's','t','u','v','
    '> w','x','y','z')
    ->         CHARACTER SET utf8,
    ->         unique (set_unique_utf8)
    -> );
Query OK, 0 rows affected (0.33 sec)

mysql> INSERT INTO C ( set_unique_utf8 ) VALUES ( '' );
Query OK, 1 row affected (0.06 sec)

mysql> INSERT INTO C ( set_unique_utf8 ) VALUES ( '' );
ERROR 1062 (23000): Duplicate entry '' for key 1
[18 Aug 2008 18:36] Valeriy Kravchuk
I can not repeat this bug any more with current 5.0-bzr on Linux:

openxs@suse:/home2/openxs/dbs/5.0> bin/mysql -uroot test
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A

Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 1
Server version: 5.0.70-debug Source distribution

Type 'help;' or '\h' for help. Type '\c' to clear the buffer.

mysql> CREATE TABLE C (
    ->         set_unique_utf8 set
    -> ('a','b','c','d','e','f','g','h','i','j','k','l','m','n','o','p','q','r','s','t','u','v','
    '> w','x','y','z')
    ->         CHARACTER SET utf8,
    ->         unique (set_unique_utf8)
    -> );
Query OK, 0 rows affected (0.03 sec)

mysql> INSERT INTO C ( set_unique_utf8 ) VALUES ( '' );
Query OK, 1 row affected (0.00 sec)

mysql> INSERT INTO C ( set_unique_utf8 ) VALUES ( '' );
ERROR 1062 (23000): Duplicate entry '' for key 1
[18 Aug 2008 18:38] Valeriy Kravchuk
It is also not repeatable any more with 5.1-bzr.
[18 Aug 2008 18:40] Valeriy Kravchuk
But it is still repeatable with 6.0.7 from bzr, it seems:

openxs@suse:/home2/openxs/dbs/6.0> bin/mysql -uroot test
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A

Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 1
Server version: 6.0.7-alpha-debug Source distribution

Type 'help;' or '\h' for help. Type '\c' to clear the buffer.

mysql> drop table C;
Query OK, 0 rows affected (0.02 sec)

mysql> CREATE TABLE C (
    ->         set_unique_utf8 set
    -> ('a','b','c','d','e','f','g','h','i','j','k','l','m','n','o','p','q','r','s','t','u','v','
    '> w','x','y','z')
    ->         CHARACTER SET utf8,
    ->         unique (set_unique_utf8)
    -> );
Query OK, 0 rows affected (0.03 sec)

mysql> INSERT INTO C ( set_unique_utf8 ) VALUES ( '' );
Query OK, 1 row affected (0.01 sec)

mysql> INSERT INTO C ( set_unique_utf8 ) VALUES ( '' );
ERROR 2013 (HY000): Lost connection to MySQL server during query
mysql> 080725 13:02:15 mysqld_safe Number of processes running now: 0
080725 13:02:15 mysqld_safe mysqld restarted
[19 Aug 2008 6:20] MySQL Verification Team
I pulled 5.1.28 from public launchpad pzr sources today and the problem is still there.  use valgrind to see it.
[1 Sep 2008 12:57] Bugs System
A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/53004

2728 Georgi Kodinov	2008-09-01
      Bug #38701: Crash in String::append when inserting duplicate empty strings an uft8 SET col
      
      When converting a value to string (to include it in an error message) MySQL was not
      initializing the buffer in the correct way. I was just resetting the length of the string
      and not setting 0 in the buffer itself.
      This, combined with a SET columns (that return empty string for a value not in the set) 
      causes warnings in retrieving the C pointer from the resulting string, that expects a
      trailing 0 (String::c_ptr()).
      Fixed by initializing the key result buffer so it always has a trailing zero.
[2 Sep 2008 13:23] Bugs System
A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/53059

2728 Georgi Kodinov	2008-09-02
      Bug #38701: Crash in String::append when inserting duplicate empty strings an uft8
      SET col
      
      When reporting a duplicate key error the server was making incorrect assumptions 
      on what the state of the value string to include in the error is.
      
      Fixed by accessing the data in this string in a "safe" way (without relying on it
      having a terminating 0).
[2 Sep 2008 13:25] Bugs System
A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/53060

2728 Georgi Kodinov	2008-09-02
      Bug #38701: Crash in String::append when inserting duplicate empty strings an uft8
      SET col
            
      When reporting a duplicate key error the server was making incorrect assumptions 
      on what the state of the value string to include in the error is.
      
      Fixed by accessing the data in this string in a "safe" way (without relying on it
      having a terminating 0).
[2 Sep 2008 13:36] Bugs System
A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/53062

2728 Georgi Kodinov	2008-09-02
      Bug #38701: Crash in String::append when inserting duplicate empty strings an uft8
      SET col
                  
      When reporting a duplicate key error the server was making incorrect assumptions 
      on what the state of the value string to include in the error is.
            
      Fixed by accessing the data in this string in a "safe" way (without relying on it
      having a terminating 0).
      
      Detected by code analysis and fixed a similar problem in reporting the foreign key
      duplicate errors.
[2 Sep 2008 13:49] Davi Arnaut
This happens to trigger a different problem on 6.0. Reported as Bug#39186
[5 Sep 2008 15:22] Bugs System
A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/53370

2681 Georgi Kodinov	2008-09-05
      Bug #38701: Crash in String::append when inserting duplicate empty strings an uft8
      SET col
                        
      When reporting a duplicate key error the server was making incorrect assumptions 
      on what the state of the value string to include in the error is.
      
      Fixed by accessing the data in this string in a "safe" way (without relying on it
      having a terminating 0).
            
      Detected by code analysis and fixed a similar problem in reporting the foreign key
      duplicate errors.
[15 Sep 2008 8:19] Bugs System
Pushed into 5.1.29  (revid:kgeorge@mysql.com-20080905152159-kzi934yt14kypudc) (version source revid:kgeorge@mysql.com-20080910094421-1i1kxv3n1bxskiqa) (pib:3)
[15 Sep 2008 18:49] Paul DuBois
Noted in 5.1.29 changelog.

When inserting a string into a duplicate-key error message, the 
server could improperly interpret the string, resulting in a crash.

Setting report to NDI pending push into 6.0.x.
[1 Oct 2008 15:54] Bugs System
Pushed into 5.1.29  (revid:kgeorge@mysql.com-20080905152159-kzi934yt14kypudc) (version source revid:kgeorge@mysql.com-20080910094421-1i1kxv3n1bxskiqa) (pib:4)
[1 Oct 2008 17:16] Paul DuBois
Setting report to NDI pending push into 6.0.x.
[17 Oct 2008 16:44] Bugs System
Pushed into 6.0.8-alpha  (revid:kgeorge@mysql.com-20080905152159-kzi934yt14kypudc) (version source revid:kpettersson@mysql.com-20080911114255-81pt7q1uvl1fkojq) (pib:5)
[17 Oct 2008 18:23] Paul DuBois
Noted in 6.0.8 changelog.
[28 Oct 2008 21:04] Bugs System
Pushed into 5.1.29-ndb-6.2.17  (revid:kgeorge@mysql.com-20080905152159-kzi934yt14kypudc) (version source revid:tomas.ulin@sun.com-20081028140209-u4emkk1xphi5tkfb) (pib:5)
[28 Oct 2008 22:23] Bugs System
Pushed into 5.1.29-ndb-6.3.19  (revid:kgeorge@mysql.com-20080905152159-kzi934yt14kypudc) (version source revid:tomas.ulin@sun.com-20081028194045-0353yg8cvd2c7dd1) (pib:5)
[1 Nov 2008 9:48] Bugs System
Pushed into 5.1.29-ndb-6.4.0  (revid:kgeorge@mysql.com-20080905152159-kzi934yt14kypudc) (version source revid:jonas@mysql.com-20081101082305-qx5a1bj0z7i8ueys) (pib:5)