Bug #47073 valgrind errs, corruption,failed repair of partition,low myisam_sort_buffer_size
Submitted: 2 Sep 2009 19:14 Modified: 18 Dec 2009 13:23
Reporter: Shane Bester (Platinum Quality Contributor) Email Updates:
Status: Closed Impact on me:
None 
Category:MySQL Server: MyISAM storage engine Severity:S3 (Non-critical)
Version:5.0.90, 5.1.37, 5.1.39 OS:Any
Assigned to: Sergey Vojtovich CPU Architecture:Any
Tags: corruption, myisam_sort_buffer_size, repair table, valgrind
Triage: Triaged: D2 (Serious) / R2 (Low) / E3 (Medium)

[2 Sep 2009 19:14] Shane Bester
Description:
when myisam_sort_buffer_size is too low, partitioned tables get corrupted during a repair table.  non-partitioned tables didn't.  There's invalid memory reads happening too:

Output of testcase (cut to fit without wrapping!):

mysql> check table `t1`;                                                
+---------+-------+----------+----------+                               
| Table   | Op    | Msg_type | Msg_text |                               
+---------+-------+----------+----------+                               
| test.t1 | check | status   | OK       |                               
+---------+-------+----------+----------+                               
1 row in set (7.78 sec)                                                 
                                                                        
mysql> repair table `t1`;                                               
+---------+--------+----------+----------------------------------------+
| Table   | Op     | Msg_type | Msg_text                               |
+---------+--------+----------+----------------------------------------+
| test.t1 | repair | error    | myisam_sort_buffer_size is too small   |
| test.t1 | repair | warning  | Number of rows changed from 0 to 14244 |
| test.t1 | repair | error    | myisam_sort_buffer_size is too small   |
| test.t1 | repair | error    | Partition p5 returned error            |
| test.t1 | repair | status   | Operation failed                       |
+---------+--------+----------+----------------------------------------+
5 rows in set (2 min 1.88 sec)                                          
                                                                        
mysql> check table `t1`;                                                
+---------+-------+----------+------------------------------------------
| Table   | Op    | Msg_type | Msg_text                                 
+---------+-------+----------+------------------------------------------
| test.t1 | check | warning  | Table is marked as crashed and last repai
| test.t1 | check | warning  | Size of indexfile is: 387072      Should 
| test.t1 | check | error    | Found 8485 keys of 0                     
| test.t1 | check | error    | Partition p5 returned error              
| test.t1 | check | error    | Corrupt                                  
+---------+-------+----------+------------------------------------------
5 rows in set (4.30 sec)                                                

5.1.37 valgrind output:

Invalid read of size 4                                     
at: reinit_io_cache (mf_iocache.c:388)                     
by: _create_index_by_sort (sort.c:209)                     
by: mi_repair_by_sort (mi_check.c:2430)                    
by: ha_myisam::repair (ha_myisam.cc:1156)                  
by: ha_myisam::enable_indexes(unsigned) (ha_myisam.cc:1455)
by: ha_myisam::end_bulk_insert() (ha_myisam.cc:1584)       
by: ha_partition::end_bulk_insert() (handler.h:1230)       
by: select_insert::send_eof() (handler.h:1230)             
by: do_select (sql_select.cc:10920)                        
by: JOIN::exec() (sql_select.cc:2199)                      
by: mysql_select (sql_select.cc:2388)                      
by: handle_select (sql_select.cc:268)                      

[Note] Retrying repair of: './test/t1' with keycache
[Note] Retrying repair of: './test/t1' with keycache
[Note] Retrying repair of: './test/t1' failed. Please try REPAIR EXTENDED or myisamchk

How to repeat:
run mysqld under valgrind to catch the valgrind errors.

---

set session myisam_sort_buffer_size=4;
drop table if exists `t1`;

create table `t1`(
`a` char(1) character set ucs2 collate ucs2_general_ci not null,
`ordc` int,unique key `a` (`a`), key `a_2` (`a`,`ordc`),
key `ordc` (`ordc`),key `a_3` (`a`)) 
engine=myisam partition by  key (`a`) partitions 8;

set @`a`:=6;
insert into `t1` values
(char(0),0),(char(1),1),(char(2),2),
(char(3),3),(char(4),4),(char(5),5);

insert ignore into `t1` select char(@`a`:=@`a`+1),@`a` from 
`t1`,`t1` `a`,`t1` `b`,`t1` `c`,`t1` `d`,`t1` `e`;

check table `t1`;
repair table `t1`;
check table `t1`;
----

Suggested fix:
o) fix that invalid memory read
o) try not to make entire table crashed. behave like unpartitioned repair does.
o) try improve the error message verbosity in the mysql error log.
[3 Sep 2009 3:32] Valeriy Kravchuk
Thank you for the bug report. Verified just as described with recent 5.1.39 from bzr on Mac OS X.
[8 Sep 2009 7:38] Sergey Vojtovich
We have two different problems here:
- REPAIR doesn't fall back to key cache, leaving table in an inconsistent state
- valgrind errors, which are coming from bulk inserts

I expect easy fixes for both problems, but it may take some time to find real cause.

Both problems should be pure MyISAM, at least I was able to repeat REPAIR issue without partitioning.
[6 Oct 2009 17:35] Bugs System
A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/85949

3146 Sergey Vojtovich	2009-10-06
      BUG#47073 - valgrind errs, corruption,failed repair of partition,
                  low myisam_sort_buffer_size
      
      Repair by sort (default) or parallel repair of a MyISAM table
      (doesn't matter partitioned or not) as well as bulk inserts
      and enable indexes some times didn't failover to repair with
      key cache.
      
      The problem was that after unsuccessful attempt, data file was
      closed. Whereas repair with key cache requires open data file.
      Fixed by reopening data file.
      
      Also fixed a valgrind warning, which may appear during repair
      by sort or parallel repair with certain myisam_sort_buffer_size
      number of rows and length of an index entry (very dependent).
     @ mysql-test/r/myisam.result
        A test case for BUG#47073.
     @ mysql-test/t/myisam.test
        A test case for BUG#47073.
     @ storage/myisam/mi_check.c
        Reopen data file, when repair by sort or parallel repair
        fails.
        
        When repair by sort is requested to rebuild data file, data file
        gets rebuilt while fixing first index. When rebuild is completed,
        info->dfile is pointing to temporary data file, original data file
        is closed.
        
        It may happen that repair has successfully fixed first index and
        rebuilt data file, but failed to fix second index. E.g.
        myisam_sort_buffer_size was big enough to fix first shorter index,
        but not enough to fix subsequent longer index.
        
        In this case we end up with info->dfile pointing to temporary file,
        which is removed and info->dfile is set to -1.
        
        Though repair by sort failed, the upper layer may still want to
        try repair with key cache. But it needs info->dfile pointing to
        valid data file.
     @ storage/myisam/sort.c
        When performing a copy of IO_CACHE structure, current_pos and
        current_end must be updated separatly to point to memory we're
        copying to (not to memory we're copying from).
        
        As t_file2 is always WRITE cache, proper members are write_pos
        and write_end accordingly.
[9 Oct 2009 16:16] Bugs System
A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/86429

3146 Sergey Vojtovich	2009-10-09
      BUG#47073 - valgrind errs, corruption,failed repair of partition,
                  low myisam_sort_buffer_size
      
      Repair by sort (default) or parallel repair of a MyISAM table
      (doesn't matter partitioned or not) as well as bulk inserts
      and enable indexes some times didn't failover to repair with
      key cache.
      
      The problem was that after unsuccessful attempt, data file was
      closed. Whereas repair with key cache requires open data file.
      Fixed by reopening data file.
      
      Also fixed a valgrind warning, which may appear during repair
      by sort or parallel repair with certain myisam_sort_buffer_size
      number of rows and length of an index entry (very dependent).
     @ mysql-test/r/myisam.result
        A test case for BUG#47073.
     @ mysql-test/t/myisam.test
        A test case for BUG#47073.
     @ storage/myisam/ha_myisam.cc
        Reverted fix for BUG25289. Not needed anymore.
     @ storage/myisam/mi_check.c
        Reopen data file, when repair by sort or parallel repair
        fails.
        
        When repair by sort is requested to rebuild data file, data file
        gets rebuilt while fixing first index. When rebuild is completed,
        info->dfile is pointing to temporary data file, original data file
        is closed.
        
        It may happen that repair has successfully fixed first index and
        rebuilt data file, but failed to fix second index. E.g.
        myisam_sort_buffer_size was big enough to fix first shorter index,
        but not enough to fix subsequent longer index.
        
        In this case we end up with info->dfile pointing to temporary file,
        which is removed and info->dfile is set to -1.
        
        Though repair by sort failed, the upper layer may still want to
        try repair with key cache. But it needs info->dfile pointing to
        valid data file.
     @ storage/myisam/sort.c
        When performing a copy of IO_CACHE structure, current_pos and
        current_end must be updated separatly to point to memory we're
        copying to (not to memory we're copying from).
        
        As t_file2 is always WRITE cache, proper members are write_pos
        and write_end accordingly.
[14 Oct 2009 14:39] Bugs System
Pushed into 5.1.41 (revid:joro@sun.com-20091014143611-cphb0enjlx6lpat1) (version source revid:satya.bn@sun.com-20091013071829-zc4c3go44j6re592) (merge vers: 5.1.40) (pib:13)
[15 Oct 2009 7:31] Bugs System
A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/86898

3170 Sergey Vojtovich	2009-10-15
      Disabled part of test for BUG#47073 until additional fix is pushed.
     @ mysql-test/r/myisam.result
        Disabled part of test for BUG#47073 until additional fix is pushed.
     @ mysql-test/t/myisam.test
        Disabled part of test for BUG#47073 until additional fix is pushed.
[15 Oct 2009 11:49] Bugs System
Pushed into 5.1.41 (revid:joro@sun.com-20091015114812-i3gm8km6gfruny5x) (version source revid:svoj@sun.com-20091015073111-t5qm270p7noyrx53) (merge vers: 5.1.41) (pib:13)
[22 Oct 2009 6:36] Bugs System
Pushed into 6.0.14-alpha (revid:alik@sun.com-20091022063126-l0qzirh9xyhp0bpc) (version source revid:alik@sun.com-20091019135554-s1pvptt6i750lfhv) (merge vers: 6.0.14-alpha) (pib:13)
[22 Oct 2009 7:08] Bugs System
Pushed into 5.5.0-beta (revid:alik@sun.com-20091022060553-znkmxm0g0gm6ckvw) (version source revid:alik@sun.com-20091019132831-5j6xs19ld16m7r8j) (merge vers: 5.5.0-beta) (pib:13)
[28 Oct 2009 22:07] Paul Dubois
Noted in 5.1.41, 5.5.0, 6.0.14 changelogs.

Repair by sort or parallel repair of MyISAM tables could fail to fail
over to repair with key cache.
[4 Nov 2009 9:26] Bugs System
Pushed into 5.1.41 (revid:joro@sun.com-20091104092152-qz96bzlf2o1japwc) (version source revid:kristofer.pettersson@sun.com-20091103162305-08l4gkeuif2ozsoj) (merge vers: 5.1.41) (pib:13)
[11 Nov 2009 6:53] Bugs System
Pushed into 6.0.14-alpha (revid:alik@sun.com-20091110093407-rw5g8dys2baqkt67) (version source revid:alik@sun.com-20091109080109-7dxapd5y5pxlu08w) (merge vers: 6.0.14-alpha) (pib:13)
[11 Nov 2009 7:01] Bugs System
Pushed into 5.5.0-beta (revid:alik@sun.com-20091109115615-nuohp02h8mdrz8m2) (version source revid:alik@sun.com-20091105092355-jzukkw6wnd7hmgrj) (merge vers: 5.5.0-beta) (pib:13)
[18 Dec 2009 10:36] Bugs System
Pushed into 5.1.41-ndb-7.1.0 (revid:jonas@mysql.com-20091218102229-64tk47xonu3dv6r6) (version source revid:jonas@mysql.com-20091218095730-26gwjidfsdw45dto) (merge vers: 5.1.41-ndb-7.1.0) (pib:15)
[18 Dec 2009 10:52] Bugs System
Pushed into 5.1.41-ndb-6.2.19 (revid:jonas@mysql.com-20091218100224-vtzr0fahhsuhjsmt) (version source revid:jonas@mysql.com-20091217101452-qwzyaig50w74xmye) (merge vers: 5.1.41-ndb-6.2.19) (pib:15)
[18 Dec 2009 11:07] Bugs System
Pushed into 5.1.41-ndb-6.3.31 (revid:jonas@mysql.com-20091218100616-75d9tek96o6ob6k0) (version source revid:jonas@mysql.com-20091217154335-290no45qdins5bwo) (merge vers: 5.1.41-ndb-6.3.31) (pib:15)
[18 Dec 2009 11:21] Bugs System
Pushed into 5.1.41-ndb-7.0.11 (revid:jonas@mysql.com-20091218101303-ga32mrnr15jsa606) (version source revid:jonas@mysql.com-20091218064304-ezreonykd9f4kelk) (merge vers: 5.1.41-ndb-7.0.11) (pib:15)
[18 Dec 2009 13:23] MC Brown
Already noted in earlier changelogs.
[24 Feb 2010 16:32] Shane Bester
this bug actually affects 5.0.90 still...