Bug #54486 assert in my_seek, concurrent DROP/CREATE SCHEMA, CREATE TABLE, REPAIR
Submitted: 14 Jun 2010 14:15 Modified: 6 Jan 2011 15:23
Reporter: Matthias Leich Email Updates:
Status: Closed Impact on me:
None 
Category:MySQL Server Severity:S3 (Non-critical)
Version:5.1.49 OS:Any
Assigned to: Dmitry Shulga CPU Architecture:Any
Tags: assert
Triage: Triaged: D1 (Critical)

[14 Jun 2010 14:15] Matthias Leich
Description:
The assert happens in mysys/my_ssek.c line 57:
  /*
      Make sure we are using a valid file descriptor!
  */
  DBUG_ASSERT(fd != -1); <-------------------
#if defined (_WIN32)
  newpos= my_win_lseek(fd, pos, whence);
#else
  newpos= lseek(fd, pos, whence);
#endif

Result on mysql-trunk-runtime revno: 3055 2010-06-12
(mysql-trunk revno: 3026 2010-05-24 shows the same effect)
----------------------------------------------------------
....
Program terminated with signal 6, Aborted.
...
#0  0x00007ffdefd5cce6 in pthread_kill () from /lib64/libpthread.so.0
#0  0x00007ffdefd5cce6 in pthread_kill () from /lib64/libpthread.so.0
#1  0x0000000000a13395 in my_write_core (sig=6)
                       at .../mysys/stacktrace.c:326
#2  0x0000000000545d16 in handle_segfault (sig=6)
                       at .../sql/mysqld.cc:2791
#3  <signal handler called>
#4  0x00007ffdeed725c5 in raise () from /lib64/libc.so.6
#5  0x00007ffdeed73bb3 in abort () from /lib64/libc.so.6
#6  0x00007ffdeed6b1e9 in __assert_fail () from /lib64/libc.so.6
#7  0x0000000000a0f957 in my_seek (fd=-1, pos=0, whence=2, MyFlags=0)
                       at .../mysys/my_seek.c:57
#8  0x00000000009f0ecb in init_io_cache (info=0x40f822a0, file=-1, cachesize=262108, type=READ_CACHE, seek_offset=0, use_async_io=1 '\001', cache_myflags=16)
                       at .../mysys/mf_iocache.c:209
#9  0x0000000000983c10 in mi_repair (param=0x40f81fc0, info=0x7ffde006da48, name=0x40f81cb0 "./testdb/t1", rep_quick=0)
                       at .../storage/myisam/mi_check.c:1557
#10 0x000000000097d3d9 in ha_myisam::repair (this=0x7ffde00704a8, thd=0x1e1ac68, param=@0x40f81fc0, do_optimize=false)
                       at .../storage/myisam/ha_myisam.cc:1057
#11 0x000000000097dbeb in ha_myisam::repair (this=0x7ffde00704a8, thd=0x1e1ac68, check_opt=0x1e1d250)
                       at .../storage/myisam/ha_myisam.cc:932
#12 0x0000000000738029 in handler::ha_repair (this=0x7ffde00704a8, thd=0x1e1ac68, check_opt=0x1e1d250)
                       at .../sql/handler.cc:3187
#13 0x0000000000657f48 in mysql_admin_table (thd=0x1e1ac68, tables=0x1e24ac0, check_opt=0x1e1d250, operator_name=0xad9515 "repair", lock_type=TL_WRITE, open_for_modify=false, no_warnings_for_error=false, extra_open_options=32,
    prepare_func=0x64a97c <prepare_for_repair>, operator_func=0x737fee <handler::ha_repair(THD*, st_ha_check_opt*)>, view_operator_func=0)
                       at .../sql/sql_table.cc:4937
#14 0x0000000000659305 in mysql_repair_table (thd=0x1e1ac68, tables=0x1e24ac0, check_opt=0x1e1d250)
                       at .../sql/sql_table.cc:5212
#15 0x00000000005d6ba5 in mysql_execute_command (thd=0x1e1ac68)
                       at .../sql/sql_parse.cc:3029
#16 0x00000000005dc5e2 in mysql_parse (thd=0x1e1ac68, inBuf=0x1e249d8 "REPAIR TABLE testdb . t1", length=24, parser_state=0x40f8c9b0)
                       at .../sql/sql_parse.cc:5889
#17 0x00000000005dd1de in dispatch_command (command=COM_QUERY, thd=0x1e1ac68, packet=0x1e1d999 "", packet_length=24)
                       at .../sql/sql_parse.cc:1114
#18 0x00000000005de701 in do_command (thd=0x1e1ac68)
                       at .../sql/sql_parse.cc:800
#19 0x00000000006a9116 in do_handle_one_connection (thd_arg=0x1e1ac68)
                       at .../sql/sql_connect.cc:1194
#20 0x00000000006a91db in handle_one_connection (arg=0x1e1ac68)
                       at .../sql/sql_connect.cc:1133
#21 0x00007ffdefd58040 in start_thread () from /lib64/libpthread.so.0
#22 0x00007ffdeee1308d in clone () from /lib64/libc.so.6
#23 0x0000000000000000 in ?? ()
...
.../sql/mysqld(my_print_stacktrace+0x32)[0xa13318]
.../sql/mysqld(handle_segfault+0x2a6)[0x545b29]
/lib64/libpthread.so.0[0x7ffdefd5fb30]
/lib64/libc.so.6(gsignal+0x35)[0x7ffdeed725c5]
/lib64/libc.so.6(abort+0x183)[0x7ffdeed73bb3]
/lib64/libc.so.6(__assert_fail+0xe9)[0x7ffdeed6b1e9]
.../sql/mysqld(my_seek+0xb7)[0xa0f957]
.../sql/mysqld(init_io_cache+0x24c)[0x9f0ecb]
.../sql/mysqld(mi_repair+0x257)[0x983c10]
.../sql/mysqld(_ZN9ha_myisam6repairEP3THDR17st_mi_check_paramb+0x573)[0x97d3d9]
.../sql/mysqld(_ZN9ha_myisam6repairEP3THDP15st_ha_check_opt+0x1c9)[0x97dbeb]
.../sql/mysqld(_ZN7handler9ha_repairEP3THDP15st_ha_check_opt+0x3b)[0x738029]
.../sql/mysqld[0x657f48]
.../sql/mysqld(_Z18mysql_repair_tableP3THDP10TABLE_LISTP15st_ha_check_opt+0xb1)[0x659305]
.../sql/mysqld(_Z21mysql_execute_commandP3THD+0x25e0)[0x5d6ba5]
.../sql/mysqld(_Z11mysql_parseP3THDPKcjP12Parser_state+0x1c7)[0x5dc5e2]
.../sql/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcj+0xaa3)[0x5dd1de]
.../sql/mysqld(_Z10do_commandP3THD+0x22b)[0x5de701]
.../sql/mysqld(_Z24do_handle_one_connectionP3THD+0x137)[0x6a9116]
.../sql/mysqld(handle_one_connection+0x2d)[0x6a91db]
/lib64/libpthread.so.0[0x7ffdefd58040]
/lib64/libc.so.6(clone+0x6d)[0x7ffdeee1308d]

How to repeat:
1.yy:
-----
query:
        CREATE DATABASE testdb |
        DROP DATABASE testdb |
        CREATE TABLE testdb . t1 ENGINE = MyISAM AS SELECT 13 |
        REPAIR TABLE testdb . t1 ;

1.zz (content is not important)
-------------------------------
$tables = {
   rows => [0, 1, 10 ],
   pk => [ 'int' , 'int auto_increment' ]
};

$fields = {
   types => [ 'int' ],
   indexes => [ 'key', undef ]
};

$data = {
   numbers => [ 'digit', 'null', undef ],
   strings => [ 'letter', 'english' ],
   blobs => [ 'data' ],
   temporals => ['date', 'year', 'null', undef ]
}

My command line:
----------------
nice -20 perl runall.pl --duration=1200 --queries=10000 \
--reporter=Deadlock,Backtrace,ErrorLog --threads=12 \
--basedir=/work2/repo/mysql-trunk-runtime \
--gendata=1.zz --mysqld=--lock-wait-timeout=1 \
--mysqld=--innodb-lock-wait-timeout=1 \
--mysqld=--log-output=table \
--mysqld=--loose-innodb-lock-wait-timeout=1 \
--grammar=1.yy --vardir=/dev/shm/var_1276507699 \
--mask-level=0 --mask=0 --seed=1
[14 Jun 2010 14:17] Matthias Leich
Protocol of the RQG run

Attachment: prt1 (application/octet-stream, text), 108.47 KiB.

[17 Jun 2010 11:39] Matthias Leich
Result on 5.1.49-debug-log
mysql-5.1 revno: 3410 2010-06-03
--------------------------------
Thread 1 (process 30415):
#0  0x00007f7923825ce6 in pthread_kill () from /lib64/libpthread.so.0
#1  0x0000000000b21a1f in my_write_core (sig=6) at stacktrace.c:329
#2  0x00000000006bd0ed in handle_segfault (sig=6) at mysqld.cc:2571
#3  <signal handler called>
#4  0x00007f7922a2e5c5 in raise () from /lib64/libc.so.6
#5  0x00007f7922a2fbb3 in abort () from /lib64/libc.so.6
#6  0x00007f7922a271e9 in __assert_fail () from /lib64/libc.so.6
#7  0x0000000000afd4a1 in my_seek (fd=-1, pos=0, whence=2, MyFlags=0) at my_seek.c:58
#8  0x0000000000afea58 in init_io_cache (info=0x44a104b0, file=-1, cachesize=262108, type=READ_CACHE, seek_offset=0, use_async_io=1 '\001', cache_myflags=16) at mf_iocache.c:209
#9  0x0000000000a3eca7 in mi_repair (param=0x44a101d0, info=0x197c118, name=0x44a0fec0 "./testdb/t1", rep_quick=0) at mi_check.c:1557
#10 0x0000000000a24721 in ha_myisam::repair (this=0x1cdd308, thd=0x1c345d8, param=@0x44a101d0, do_optimize=false) at ha_myisam.cc:1156
#11 0x0000000000a24f34 in ha_myisam::repair (this=0x1cdd308, thd=0x1c345d8, check_opt=0x1c36b48) at ha_myisam.cc:1031
#12 0x000000000080ad7f in handler::ha_repair (this=0x1cdd308, thd=0x1c345d8, check_opt=0x1c36b48) at handler.cc:3158
#13 0x000000000083a267 in mysql_admin_table (thd=0x1c345d8, tables=0x1c165f0, check_opt=0x1c36b48, operator_name=0xca8365 "repair", lock_type=TL_WRITE, open_for_modify=false, no_warnings_for_error=false, extra_open_options=32,
    prepare_func=0x829761 <prepare_for_repair>, operator_func=0x80ad44 <handler::ha_repair(THD*, st_ha_check_opt*)>, view_operator_func=0) at sql_table.cc:4810
#14 0x000000000083b684 in mysql_repair_table (thd=0x1c345d8, tables=0x1c165f0, check_opt=0x1c36b48) at sql_table.cc:5071
#15 0x00000000006d15c7 in mysql_execute_command (thd=0x1c345d8) at sql_parse.cc:3006
#16 0x00000000006d81ac in mysql_parse (thd=0x1c345d8, inBuf=0x1c16508 "REPAIR TABLE testdb . t1", length=24, found_semicolon=0x44a1aee0) at sql_parse.cc:5994
#17 0x00000000006d9034 in dispatch_command (command=COM_QUERY, thd=0x1c345d8, packet=0x1c36fa9 "", packet_length=24) at sql_parse.cc:1241
#18 0x00000000006da502 in do_command (thd=0x1c345d8) at sql_parse.cc:882
#19 0x00000000006c688c in handle_one_connection (arg=0x1c345d8) at sql_connect.cc:1134
#20 0x00007f7923821040 in start_thread () from /lib64/libpthread.so.0
#21 0x00007f7922acf08d in clone () from /lib64/libc.so.6
#22 0x0000000000000000 in ?? ()
[1 Sep 2010 9:27] Jon Olav Hauglid
Not able to repeat this bug in 5.5 (5.5-runtime tree).
Still able to repeat it in 5.1.
[1 Nov 2010 14:54] Jon Olav Hauglid
Editing the version tag since this bug is not repeatable in 5.5.
Still repeatable in 5.1 however.
[19 Nov 2010 5:45] Bugs System
A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/124353

3511 Dmitry Shulga	2010-11-19
      Fixed bug#54486 - assert in my_seek, concurrent
      DROP/CREATE SCHEMA, CREATE TABLE, REPAIR.
      
      The cause of assert was concurrent execution of
      DROP DATABASE and REPAIR TABLE where first statement
      deleted table's file .TMD at the same time when
      REPAIR TABLE statement tried replace
      old file by temporary file that has just been removed.
      
      Additionally was fixed trouble when DROP TABLE try delete
      all files belong to table being dropped at the same time
      when REPAIR TABLE statement has just deleted .TMD file.
     @ sql/sql_db.cc
        mysql_rm_known_files() modified: ignore possible errors
        when trying delete all table's files. Such aggressigve 
        algorithm permits skip already deleted (in another thread)
        files.
     @ storage/myisam/mi_check.c
        mi_repair() was modified: set param->retry_repair= 0
        in order to don't call following failover procedure
        in ha_myisam::repair() in case of error in access to
        .TMD file.
[29 Nov 2010 10:08] Bugs System
A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/125310

3511 Dmitry Shulga	2010-11-29
      Fixed bug#54486 - assert in my_seek, concurrent
      DROP/CREATE SCHEMA, CREATE TABLE, REPAIR.
      
      The cause of assert was concurrent execution of
      DROP DATABASE and REPAIR TABLE where first statement
      deleted table's file .TMD at the same time when
      REPAIR TABLE statement tried replace
      old file by temporary file that has just been removed.
      
      Additionally was fixed trouble when DROP TABLE try delete
      all files belong to table being dropped at the same time
      when REPAIR TABLE statement has just deleted .TMD file.
     @ sql/sql_db.cc
        mysql_rm_known_files() modified: ignore possible errors
        when trying delete all table's files. Such aggressigve 
        algorithm permits skip already deleted (in another thread)
        files.
     @ storage/myisam/mi_check.c
        mi_repair() was modified: set param->retry_repair= 0
        in order to don't call following failover procedure
        in ha_myisam::repair(); set an error in diagnostic area in
        case of error in access to .TMD file.
[3 Dec 2010 14:10] Bugs System
A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/125939

3511 Dmitry Shulga	2010-12-03
      Fixed bug#54486 - assert in my_seek, concurrent
      DROP/CREATE SCHEMA, CREATE TABLE, REPAIR.
      
      The cause of assert was concurrent execution of
      DROP DATABASE and REPAIR TABLE where first statement
      deleted table's file .TMD at the same time when
      REPAIR TABLE statement tried replace
      old file by temporary file that has just been removed.
      
      Additionally was fixed trouble when DROP TABLE try delete
      all files belong to table being dropped at the same time
      when REPAIR TABLE statement has just deleted .TMD file.
      
      Regression test isn't added because for such one need insert 
      synchonization point (DEBUG_SYNC) to function my_redel() defined in
      mysys/my_redel.c. DEBUG_SYNC is defined in sql/debug_sync.h.
      So, in order to add this one we need add additional dependency,
      i.e. mysys/my_redel.c will depend on sql.debug_sync.h, that is wrong.
     @ sql/sql_db.cc
        mysql_rm_known_files() modified: ignore possible errors
        when trying delete all table's files. Such aggressigve 
        algorithm permits skip already deleted (in another thread)
        files.
     @ storage/myisam/mi_check.c
        mi_repair() was modified: set param->retry_repair= 0
        in order to don't call following failover procedure
        in ha_myisam::repair(); set an error in diagnostic area in
        case of error in access to .TMD file.
[9 Dec 2010 15:49] Bugs System
A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/126441

3514 Dmitry Shulga	2010-12-09
      Fixed bug#54486 - assert in my_seek, concurrent
      DROP/CREATE SCHEMA, CREATE TABLE, REPAIR.
      
      The cause of assert was concurrent execution of
      DROP DATABASE and REPAIR TABLE where first statement
      deleted table's file .TMD at the same time as
      REPAIR TABLE tried to read file details from the old file
      that was just removed.
      
      Additionally was fixed trouble when DROP TABLE try delete
      all files belong to table being dropped at the same time
      when REPAIR TABLE statement has just deleted .TMD file.
      
      No regression test added because this would require adding a
      sync point to mysys/my_redel.c. Since this bug is not present in
      5.5+, adding test coverage was considered unnecessary.
      The patch has been verified using RQG testing.
     @ sql/sql_db.cc
        mysql_rm_known_files() modified: ignore possible errors
        when trying delete all table's files. Such aggressive 
        algorithm permits skip already deleted (in another thread)
        files.
     @ storage/myisam/mi_check.c
        mi_repair() was modified: set param->retry_repair= 0
        in order to don't call following failover procedure
        in ha_myisam::repair().
[10 Dec 2010 7:50] Bugs System
A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/126502

3515 Dmitry Shulga	2010-12-10
      Fixed bug#54486 - assert in my_seek, concurrent
      DROP/CREATE SCHEMA, CREATE TABLE, REPAIR.
      
      The cause of assert was concurrent execution of
      DROP DATABASE and REPAIR TABLE where first statement
      deleted table's file .TMD at the same time as
      REPAIR TABLE tried to read file details from the old file
      that was just removed.
      
      Additionally was fixed trouble when DROP TABLE try delete
      all files belong to table being dropped at the same time
      when REPAIR TABLE statement has just deleted .TMD file.
      
      No regression test added because this would require adding a
      sync point to mysys/my_redel.c. Since this bug is not present in
      5.5+, adding test coverage was considered unnecessary.
      The patch has been verified using RQG testing.
     @ sql/sql_db.cc
        mysql_rm_known_files() modified: ignore possible ENOENT error
        when trying delete all table's files. Such aggressive 
        algorithm permits skip already deleted (in another thread)
        files.
        
        Installation of Drop_table_error_handler as internal error handler
        moved from mysql_rm_db() to mysql_rm_knowns_files() near to place
        where source of possible errors (call to mysql_rm_table_part2) located.
     @ storage/myisam/mi_check.c
        mi_repair() was modified: set param->retry_repair= 0
        in order to don't call following failover procedure
        in ha_myisam::repair().
[10 Dec 2010 11:35] Bugs System
A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/126517

3183 Dmitry Shulga	2010-12-10 [merge]
      Manual merge from mysql-5.1-bugteam for bug#54486.
[10 Dec 2010 12:57] Bugs System
A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/126524

3419 Dmitry Shulga	2010-12-10 [merge]
      Merge from mysql-5.5-bugteam for bug#54486.
[10 Dec 2010 12:59] Dmitry Shulga
Patch was pushed to mysql-5.1-bugteam, mysql-5.5-bugteam, mysql-trunk-bugfixing.
[17 Dec 2010 12:48] Bugs System
Pushed into mysql-5.1 5.1.55 (revid:georgi.kodinov@oracle.com-20101217124435-9imm43geck5u55qw) (version source revid:dmitry.shulga@oracle.com-20101210074850-0cttphw0aga0j8km) (merge vers: 5.1.55) (pib:24)
[17 Dec 2010 12:52] Bugs System
Pushed into mysql-5.5 5.5.9 (revid:georgi.kodinov@oracle.com-20101217124733-p1ivu6higouawv8l) (version source revid:dmitry.shulga@oracle.com-20101210113258-5kuldljyahvu390t) (merge vers: 5.5.8) (pib:24)
[17 Dec 2010 12:56] Bugs System
Pushed into mysql-trunk 5.6.1 (revid:georgi.kodinov@oracle.com-20101217125013-y8pb3az32rtbplc9) (version source revid:dmitry.shulga@oracle.com-20101210125255-6pb95vnvpowxphti) (merge vers: 5.6.1) (pib:24)
[6 Jan 2011 15:23] Paul Dubois
Noted in 5.1.55, 5.5.9 changelogs.

An assertion could be raised during concurrent execution of DROP
DATABASE and REPAIR TABLE if the drop deleted a table's .TMD file at
the same time the repair tried to read details from the old file that
was just removed.

A problem could also occur when DROP TABLE tried to remove all files 
belonging to a table at the same time REPAIR TABLE had just deleted
the table's .TMD file.