MySQL Bugs: #41541: Valgrind warnings on packed MyISAM table

Bug #41541	Valgrind warnings on packed MyISAM table
Submitted:	17 Dec 2008 12:25	Modified:	13 May 2009 23:28
Reporter:	Sergei Glukhov	Email Updates:
Status:	Closed	Impact on me:	None
Category:	MySQL Server: MyISAM storage engine	Severity:	S3 (Non-critical)
Version:	5.0, 5.1, 6.0	OS:	Any
Assigned to:	Satya B	CPU Architecture:	Any
Tags:	pushbuild, sporadic, test failure, valgrind

Description:
If a table has VARCHAR or TEXT field and packed using myisampack tool
then valgrind produces warnings on the table opening:

==7743== Thread 15:
==7743== Invalid read of size 1
==7743==    at 0x8571BE2: fill_buffer (mi_packrec.c:1448)
==7743==    by 0x8571B88: fill_and_get_bits (mi_packrec.c:1419)
==7743==    by 0x856F5E5: read_huff_table (mi_packrec.c:384)
==7743==    by 0x856EC60: _mi_read_pack_info (mi_packrec.c:249)
==7743==    by 0x855C60C: mi_open (mi_open.c:552)
==7743==    by 0x85530AD: ha_myisam::open(char const*, int, unsigned) (ha_myisam.cc:616)
==7743==    by 0x843583F: handler::ha_open(TABLE*, char const*, int, int) (handler.cc:2018)
==7743==    by 0x836AA8A: open_table_from_share(THD*, TABLE_SHARE*, char const*, unsigned, unsigned, uns
==7743==    by 0x83608AE: open_table(THD*, TABLE_LIST*, st_mem_root*, enum_open_table_action*, unsigned)
==7743==    by 0x836168A: open_tables(THD*, TABLE_LIST**, unsigned*, unsigned) (sql_base.cc:3717)
==7743==    by 0x8361D79: open_and_lock_tables_derived(THD*, TABLE_LIST*, bool, unsigned) (sql_base.cc:4
==7743==    by 0x831CC6E: open_and_lock_tables(THD*, TABLE_LIST*) (mysql_priv.h:1610)
==7743==    by 0x830FEB4: _ZL21execute_sqlcom_selectP3THDP10TABLE_LIST (sql_parse.cc:4714)
==7743==    by 0x8311B5B: mysql_execute_command(THD*) (sql_parse.cc:2061)
==7743==    by 0x831A29E: mysql_parse(THD*, char const*, unsigned, char const**) (sql_parse.cc:5737)
==7743==    by 0x831AC6C: dispatch_command(enum_server_command, THD*, char*, unsigned) (sql_parse.cc:100
==7743==  Address 0x10e72c8d is 2 bytes after a block of size 4,155 alloc'd
==7743==    at 0x4006AEE: malloc (vg_replace_malloc.c:207)
==7743==    by 0x890A58F: my_malloc (my_malloc.c:34)
==7743==    by 0x856E820: _mi_read_pack_info (mi_packrec.c:213)
==7743==    by 0x855C60C: mi_open (mi_open.c:552)
==7743==    by 0x85530AD: ha_myisam::open(char const*, int, unsigned) (ha_myisam.cc:616)
==7743==    by 0x843583F: handler::ha_open(TABLE*, char const*, int, int) (handler.cc:2018)
==7743==    by 0x836AA8A: open_table_from_share(THD*, TABLE_SHARE*, char const*, unsigned, unsigned, uns
==7743==    by 0x83608AE: open_table(THD*, TABLE_LIST*, st_mem_root*, enum_open_table_action*, unsigned)
==7743==    by 0x836168A: open_tables(THD*, TABLE_LIST**, unsigned*, unsigned) (sql_base.cc:3717)
==7743==    by 0x8361D79: open_and_lock_tables_derived(THD*, TABLE_LIST*, bool, unsigned) (sql_base.cc:4
==7743==    by 0x831CC6E: open_and_lock_tables(THD*, TABLE_LIST*) (mysql_priv.h:1610)
==7743==    by 0x830FEB4: _ZL21execute_sqlcom_selectP3THDP10TABLE_LIST (sql_parse.cc:4714)
==7743==    by 0x8311B5B: mysql_execute_command(THD*) (sql_parse.cc:2061)
==7743==    by 0x831A29E: mysql_parse(THD*, char const*, unsigned, char const**) (sql_parse.cc:5737)
==7743==    by 0x831AC6C: dispatch_command(enum_server_command, THD*, char*, unsigned) (sql_parse.cc:100
==7743==    by 0x831BEFF: do_command(THD*) (sql_parse.cc:689)

repeatable in 5.0,5.1,6.0 versions

How to repeat:
--disable_warnings
drop table if exists t1;
--enable_warnings
create table t1(f1 int, f2 varchar(255));
#
# There is valgrind error with TEXT field also
#create table t1(f1 int, f2 text);
insert into t1 values(1, 'foo'), (2, 'bar');
insert into t1 select * from t1;
insert into t1 select * from t1;
insert into t1 select * from t1;
insert into t1 select * from t1;
insert into t1 select * from t1;
insert into t1 select * from t1;
insert into t1 select * from t1;
insert into t1 select * from t1;
insert into t1 select * from t1;
insert into t1 select * from t1;
insert into t1 select * from t1;
insert into t1 select * from t1;
flush tables;
--exec $MYISAMPACK $MYSQLTEST_VARDIR/master-data/test/t1
select count(*) from t1;
drop table t1;

Thank you for the bug report. Verified as described:

==25497== Invalid read of size 1
==25497==    at 0x84F5DF: fill_buffer (mi_packrec.c:1446)
==25497==    by 0x84F623: fill_and_get_bits (mi_packrec.c:1417)
==25497==    by 0x85182A: _mi_read_pack_info (mi_packrec.c:382)
==25497==    by 0x845B91: mi_open (mi_open.c:498)
==25497==    by 0x6B4942: ha_myisam::open(char const*, int, unsigned) (ha_myisam.cc:633)
==25497==    by 0x6AEE61: handler::ha_open(char const*, int, int) (handler.cc:1414)
==25497==    by 0x62A561: openfrm(THD*, char const*, char const*, unsigned, unsigned, unsigned, st_table*) (table.cc:929)
==25497==    by 0x61E92B: _ZL17open_unireg_entryP3THDP8st_tablePKcS4_S4_P10TABLE_LISTP11st_mem_rootj (sql_base.cc:2457)
==25497==    by 0x621432: open_table(THD*, TABLE_LIST*, st_mem_root*, bool*, unsigned) (sql_base.cc:1922)
==25497==    by 0x621BDE: open_tables(THD*, TABLE_LIST**, unsigned*, unsigned) (sql_base.cc:2761)
==25497==    by 0x621F67: open_normal_and_derived_tables(THD*, TABLE_LIST*, unsigned) (sql_base.cc:3101)
==25497==    by 0x6F6A47: mysqld_list_fields(THD*, TABLE_LIST*, char const*) (sql_show.cc:587)
==25497==  Address 0xc8b9a03 is 0 bytes after a block of size 4,155 alloc'd
==25497==    at 0x4C265AE: malloc (vg_replace_malloc.c:207)
==25497==    by 0x88E1D2: my_malloc (my_malloc.c:34)
==25497==    by 0x850E85: _mi_read_pack_info (mi_packrec.c:211)
==25497==    by 0x845B91: mi_open (mi_open.c:498)
==25497==    by 0x6B4942: ha_myisam::open(char const*, int, unsigned) (ha_myisam.cc:633)
==25497==    by 0x6AEE61: handler::ha_open(char const*, int, int) (handler.cc:1414)
==25497==    by 0x62A561: openfrm(THD*, char const*, char const*, unsigned, unsigned, unsigned, st_table*) (table.cc:929)
==25497==    by 0x61E92B: _ZL17open_unireg_entryP3THDP8st_tablePKcS4_S4_P10TABLE_LISTP11st_mem_rootj (sql_base.cc:2457)
==25497==    by 0x621432: open_table(THD*, TABLE_LIST*, st_mem_root*, bool*, unsigned) (sql_base.cc:1922)
==25497==    by 0x621BDE: open_tables(THD*, TABLE_LIST**, unsigned*, unsigned) (sql_base.cc:2761)
==25497==    by 0x621F67: open_normal_and_derived_tables(THD*, TABLE_LIST*, unsigned) (sql_base.cc:3101)
==25497==    by 0x6F6A47: mysqld_list_fields(THD*, TABLE_LIST*, char const*) (sql_show.cc:587)
==25497== 
==25497== 1 errors in context 2 of 6:
==25497== Invalid read of size 1
==25497==    at 0x84F5D6: fill_buffer (mi_packrec.c:1446)
==25497==    by 0x84F623: fill_and_get_bits (mi_packrec.c:1417)
==25497==    by 0x85182A: _mi_read_pack_info (mi_packrec.c:382)
==25497==    by 0x845B91: mi_open (mi_open.c:498)
==25497==    by 0x6B4942: ha_myisam::open(char const*, int, unsigned) (ha_myisam.cc:633)
==25497==    by 0x6AEE61: handler::ha_open(char const*, int, int) (handler.cc:1414)
==25497==    by 0x62A561: openfrm(THD*, char const*, char const*,

<cut>

A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/67528

2754 Satya B	2009-02-25
      Fix for BUG#41541 - Valgrind warnings on packed MyISAM table
      
      myisampack tool with valgrind throws "invalid read size" errors 
      when a table with varchar or text field exist
      
      This happens because when we try to read a record into the buffer
      we alway assume that the remaing buffer to read is always equal 
      to word size(4 or 8 or 2 bytes) we read. Sometimes we have buffer
      size less than word size and trying to read the entire word size
      will end up in valgrind errors
      
      Fixed by reading byte by byte when we detect the buffer size is 
      less than the word size
      modified:
        mysql-test/r/myisampack.result
        mysql-test/t/myisampack.test
        storage/myisam/mi_packrec.c

Please fix some minor problems. See email.

A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/68103

2754 Satya B	2009-03-03
      Fix for BUG#41541 - Valgrind warnings on packed MyISAM table
      
      After the table is compressed by the myisampack utility,
      opening the table by the server produces valgrind warnings.
      
      This happens because when we try to read a record into the buffer
      we alway assume that the remaining buffer to read is always equal 
      to word size(4 or 8 or 2 bytes) we read. Sometimes we have 
      remaining buffer size less than word size and trying to read the 
      entire word size will end up in valgrind errors
      
      Fixed by reading byte by byte when we detect the remaining buffer 
      size is less than the word size
      modified:
        mysql-test/r/myisampack.result
        mysql-test/t/myisampack.test
        storage/myisam/mi_packrec.c

A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/68164

2754 Satya B	2009-03-04
      Fix for BUG#41541 - Valgrind warnings on packed MyISAM table
            
      After the table is compressed by the myisampack utility,
      opening the table by the server produces valgrind warnings.
      
      This happens because when we try to read a record into the buffer
      we alway assume that the remaining buffer to read is always equal 
      to word size(4 or 8 or 2 bytes) we read. Sometimes we have 
      remaining buffer size less than word size and trying to read the 
      entire word size will end up in valgrind errors.
            
      Fixed by reading byte by byte when we detect the remaining buffer 
      size is less than the word size.
      modified:
        mysql-test/r/myisampack.result
        mysql-test/t/myisampack.test
        storage/myisam/mi_packrec.c

Bug#43577 has been marked as a duplicate of this one.

Approved with wishlist.

A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/70319

2743 Satya B	2009-03-25
      Fix for BUG#41541 - Valgrind warnings on packed MyISAM table
                  
      After the table is compressed by the myisampack utility,
      opening the table by the server produces valgrind warnings.
            
      This happens because when we try to read a record into the buffer
      we alway assume that the remaining buffer to read is always equal 
      to word size(4 or 8 or 2 bytes) we read. Sometimes we have 
      remaining buffer size less than word size and trying to read the 
      entire word size will end up in valgrind errors.
                  
      Fixed by reading byte by byte when we detect the remaining buffer 
      size is less than the word size.
      modified:
        myisam/mi_packrec.c
        mysql-test/r/myisampack.result
        mysql-test/t/myisampack.test

Pushed into 5.0.82 (revid:davi.arnaut@sun.com-20090505184158-dvmedh8n472y8np5) (version source revid:davi.arnaut@sun.com-20090505184158-dvmedh8n472y8np5) (merge vers: 5.0.82) (pib:6)

Pushed into 5.1.35 (revid:davi.arnaut@sun.com-20090505190206-9xmh7dlc6kom8exp) (version source revid:davi.arnaut@sun.com-20090505190206-9xmh7dlc6kom8exp) (merge vers: 5.1.35) (pib:6)

Pushed into 6.0.12-alpha (revid:svoj@sun.com-20090506125450-yokcmvqf2g7jhujq) (version source revid:horst@mysql.com-20090327184517-25eq077q2beocs6y) (merge vers: 6.0.11-alpha) (pib:6)

Noted in 5.0.82, 5.1.35, 6.0.12 changelogs.

Compressing a table with the myisampack utility caused the server to
produce Valgrind warnings when it opened the table.

Pushed into 5.1.35-ndb-6.3.26 (revid:jonas@mysql.com-20090615074202-0r5r2jmi83tww6sf) (version source revid:jonas@mysql.com-20090615070837-9pccutgc7repvb4d) (merge vers: 5.1.35-ndb-6.3.26) (pib:6)

Pushed into 5.1.35-ndb-7.0.7 (revid:jonas@mysql.com-20090615074335-9hcltksp5cu5fucn) (version source revid:jonas@mysql.com-20090615072714-rmfkvrbbipd9r32c) (merge vers: 5.1.35-ndb-7.0.7) (pib:6)

Pushed into 5.1.35-ndb-6.2.19 (revid:jonas@mysql.com-20090615061520-sq7ds4yw299ggugm) (version source revid:jonas@mysql.com-20090615054654-ebgpz7elwu1xj36j) (merge vers: 5.1.35-ndb-6.2.19) (pib:6)