Bug #44811 Tests with utf8 charset fail with ibmdb2i on 64bit MySQL
Submitted: 12 May 2009 4:59 Modified: 14 Jul 2009 15:20
Reporter: Nidhi Shrotriya Email Updates:
Status: Closed Impact on me:
None 
Category:MySQL Server: DB2SE for IBM i Severity:S3 (Non-critical)
Version:5.1.34 OS:IBM i (5.4 & 6.1)
Assigned to: CPU Architecture:Any

[12 May 2009 4:59] Nidhi Shrotriya
Description:
1) ctype_utf8.test(test modified by IBM) fails with
mysqltest: In included file "/home/nidhi/mysql-5.1.33-i5os-power-64bit/mysql-test/t/ctype_utf8-2.sql": At line 3: query 'insert into t1 select repeat('�',87)' failed: 1296: Got error 2021 'See message CPF5035 in joblog for job 177510/QUSER/QSQSRVR.' from IBMDB2I

This happens only with 64-bit build of MySQL when using ibmdb2i. 

Looks like a storage space issue as I tried to do the above repeat() with different nos. and it passes with
insert into t1 select repeat('�',86);
fails with
insert into t1 select repeat('�',87);
with the error CPF5035 as below.
mysql> source mysql-test/t/ctype_utf8-2.sql;
Query OK, 0 rows affected (0.07 sec)

Query OK, 1 row affected (0.01 sec)
Records: 1  Duplicates: 0  Warnings: 0

Query OK, 1 row affected (0.00 sec)
Records: 1  Duplicates: 0  Warnings: 0

+------------+-----------------+
| length(s1) | char_length(s1) |
+------------+-----------------+
|        100 |             100 |
|        172 |              86 |
+------------+-----------------+
2 rows in set (0.00 sec)

Query OK, 0 rows affected (0.03 sec)

mysql> source mysql-test/t/ctype_utf8-2.sql;
Query OK, 0 rows affected (0.06 sec)

Query OK, 1 row affected (0.01 sec)
Records: 1  Duplicates: 0  Warnings: 0

ERROR 1296 (HY000): Got error 2021 'See message CPF5035 in joblog for job 175060/QUSER/QSQSRVR.' from IBMDB2I
+------------+-----------------+
| length(s1) | char_length(s1) |
+------------+-----------------+
|        100 |             100 |
+------------+-----------------+
1 row in set (0.00 sec)

Query OK, 0 rows affected (0.03 sec)

If character taking 3 bytes may be 86*3=258 > 256+1....something to do with that ??

2) ibmdb2i_rir.test (test added by IBM) fails as below
mysqltest: At line 429: query 'insert into t1 values ('さしすせそかきくけこあいうえお')' failed: 1296: Got error 2021 'See message CPF5035 in joblog for job 177510/QUSER/QSQSRVR.' from IBMDB2I

Passes with MyISAM/Innodb.

How to repeat:
Attaching .sql files to reproduce with ibmdb2i and 64bit build of mysql.
[12 May 2009 5:00] Nidhi Shrotriya
Sql file for ctype_utf8.test

Attachment: ctype_utf8-2.sql (text/x-sql), 654 bytes.

[12 May 2009 5:01] Nidhi Shrotriya
Sql file for ibmdb2i_rir.test

Attachment: ibmdb2i_rir_1.sql (text/x-sql), 474 bytes.

[29 May 2009 9:34] Bugs System
A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/75223

2918 Narayanan V	2009-05-29
      Bug#44811 Tests with utf8 charset fail with ibmdb2i on 64bit MySQL
      
      wmemset was being used to fill the row buffers.
      wmemset was intended  to  fill the buffer with
      16-bit UCS2 pad values.  However,  the  64-bit
      version of wmemset uses 32-bit wide characters
      and thus filled the buffer incorrectly. In some
      cases, the null  byte  map would be overwritten,
      causing ctype_utf8.test and ibmdb2i_rir.test to
      fail, giving the error message CPF5035.
      
      This patch eliminates the use of wmemset to fill
      the row buffer. wmemset has  been  replaced with
      memset16, which always  fills  memory with 16-bit
      values.
     @ storage/ibmdb2i/db2i_conversion.cc
        Bug#44811 Tests with utf8 charset fail with ibmdb2i on 64bit MySQL
        
        Eliminate the use of wmemset to fill
        the row buffer. Replace wmemset with
        memset16, which always  fills  memory
        with 16-bit values.
     @ storage/ibmdb2i/db2i_misc.h
        Bug#44811 Tests with utf8 charset fail with ibmdb2i on 64bit MySQL
        
        Eliminate the use of wmemset to fill
        the row buffer. Replace wmemset with
        memset16, which always  fills  memory
        with 16-bit values.
[29 May 2009 10:01] Bugs System
A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/75224

2918 Narayanan V	2009-05-29
      Bug#44811 Tests with utf8 charset fail with ibmdb2i on 64bit MySQL
      
      wmemset was being used to fill the row buffers.
      wmemset was intended  to  fill the buffer with
      16-bit UCS2 pad values.  However,  the  64-bit
      version of wmemset uses 32-bit wide characters
      and thus filled the buffer incorrectly. In some
      cases, the null  byte  map would be overwritten,
      causing ctype_utf8.test and ibmdb2i_rir.test to
      fail, giving the error message CPF5035.
      
      This patch eliminates the use of wmemset to fill
      the row buffer. wmemset has  been  replaced with
      memset16, which always  fills  memory with 16-bit
      values.
     @ storage/ibmdb2i/db2i_conversion.cc
        Bug#44811 Tests with utf8 charset fail with ibmdb2i on 64bit MySQL
        
        wmemset has  been  replaced with memset16,
        which always  fills  memory with 16-bit
        values.
     @ storage/ibmdb2i/db2i_misc.h
        Bug#44811 Tests with utf8 charset fail with ibmdb2i on 64bit MySQL
        
        wmemset has  been  replaced with memset16,
        which always  fills  memory with 16-bit
        values.
[16 Jun 2009 11:03] Bugs System
Pushed into 5.1.36 (revid:joro@sun.com-20090616102155-3zhezogudt4uxdyn) (version source revid:azundris@mysql.com-20090529164935-xe3dceff53d7pywb) (merge vers: 5.1.36) (pib:6)
[17 Jun 2009 19:28] Bugs System
Pushed into 5.4.4-alpha (revid:alik@sun.com-20090616183122-chjzbaa30qopdra9) (version source revid:azundris@mysql.com-20090529170733-wxq9j0idmss9rllz) (merge vers: 6.0.12-alpha) (pib:11)
[14 Jul 2009 15:20] MC Brown
A note has been added to the 5.1.36 and 5.4.4 changelog: 

Running queries on tables with the IBMDB2I storage engine using the utf8 character would fail when using the 64-bit version of MySQL
[12 Aug 2009 22:26] Paul DuBois
Noted in 5.4.2 changelog because next 5.4 version will be 5.4.2 and not 5.4.4.
[15 Aug 2009 1:41] Paul DuBois
Ignore previous comment about 5.4.2.
[26 Aug 2009 13:46] Bugs System
Pushed into 5.1.37-ndb-7.0.8 (revid:jonas@mysql.com-20090826132541-yablppc59e3yb54l) (version source revid:jonas@mysql.com-20090826132541-yablppc59e3yb54l) (merge vers: 5.1.37-ndb-7.0.8) (pib:11)
[26 Aug 2009 13:46] Bugs System
Pushed into 5.1.37-ndb-6.3.27 (revid:jonas@mysql.com-20090826105955-bkj027t47gfbamnc) (version source revid:jonas@mysql.com-20090826105955-bkj027t47gfbamnc) (merge vers: 5.1.37-ndb-6.3.27) (pib:11)
[26 Aug 2009 13:48] Bugs System
Pushed into 5.1.37-ndb-6.2.19 (revid:jonas@mysql.com-20090825194404-37rtosk049t9koc4) (version source revid:jonas@mysql.com-20090825194404-37rtosk049t9koc4) (merge vers: 5.1.37-ndb-6.2.19) (pib:11)
[27 Aug 2009 16:33] Bugs System
Pushed into 5.1.35-ndb-7.1.0 (revid:magnus.blaudd@sun.com-20090827163030-6o3kk6r2oua159hr) (version source revid:jonas@mysql.com-20090826132541-yablppc59e3yb54l) (merge vers: 5.1.37-ndb-7.0.8) (pib:11)
[7 Oct 2009 19:14] Paul DuBois
The 5.4 fix has been pushed to 5.4.2.