Bug #41725 slave crashes when inserting into temporary table after stop/start slave
Submitted: 24 Dec 2008 3:52 Modified: 24 Jun 2009 14:17
Reporter: Shane Bester (Platinum Quality Contributor) Email Updates:
Status: Closed Impact on me:
None 
Category:MySQL Server: Replication Severity:S1 (Critical)
Version:5.0.70, 5.0.72, 5.0.74,5.1.30, 5.1.32, 6.0.8 OS:Any
Assigned to: Luis Soares CPU Architecture:Any
Triage: Triaged: D1 (Critical) / R2 (Low) / E3 (Medium)

[24 Dec 2008 3:52] Shane Bester
Description:
when the slave is replication many temporary tables and inserts into those, running 'stop slave' then 'start slave' can crash with this stack trace:

mysqld-nt.exe!sp_rcontext::find_handler  Line 200
mysqld-nt.exe!sp_rcontext::handle_error  Line 313
mysqld-nt.exe!push_warning
mysqld-nt.exe!push_warning_printf
mysqld-nt.exe!Field::set_warning
mysqld-nt.exe!Field_longstr::report_if_important_data
mysqld-nt.exe!Field_string::store
mysqld-nt.exe!fill_record
mysqld-nt.exe!fill_record_n_invoke_before_triggers
mysqld-nt.exe!mysql_insert
mysqld-nt.exe!mysql_execute_command
mysqld-nt.exe!mysql_parse
mysqld-nt.exe!Query_log_event::exec_event
mysqld-nt.exe!exec_relay_log_event
mysqld-nt.exe!handle_slave_sql
mysqld-nt.exe!pthread_start
mysqld-nt.exe!_callthreadstart
mysqld-nt.exe!_threadstart

Suddenly stopping a slave with open tmp tables can break replication, but a crash should *not* happen.

How to repeat:
will upload a testcase later.  I can reproduce it about 80% of the time.

on a very fast machine, replicate these continuously on the master:

drop temporary table if exists t1
create temporary table t1 ...
100x insert into t1 ...

looks like warning is generated by insert, so try using too long values for some fields..

while slave is replicating at full speed:

stop slave;
start slave;
select sleep(1);
stop slave;
start slave;
...
[24 Dec 2008 6:25] Shane Bester
5.1.30 crashed the same.

how to repeat:
--------------
setup a replication between master/slave.
on master, run this:

delimiter //
drop procedure if exists `p1`//
create procedure `p1`(`num` int unsigned)
begin
  declare `i` int unsigned default '0';
  declare `j` int unsigned default '0';
  repeat
    drop temporary table if exists `t1`;
    create temporary table `t1`(`a` tinyint,`b` char(1))engine=myisam;
    set `j`=0;
    repeat
      insert into `t1` set `a`=1000*rand(),`b`=rand();
      set `j`=`j`+1;
    until `j`>100 end repeat;
    set `i`=`i`+1;
  until `i`>`num` end repeat;    
end//

delimiter ;
call `p1`(100000);

on slave, do this 

stop slave;start slave;
stop slave;start slave;
stop slave;start slave;
stop slave;start slave;
etc.
[29 Dec 2008 12:57] Shane Bester
5.0.74 debug binary causes an assertion with this testcase:

Assertion failed: table->in_use == _current_thd(), file .\field.cc, line 5920

588FE5    mysqld-debug.exe!my_sigabrt_handler()[mysqld.cc:1847]
A29EE1    mysqld-debug.exe!raise()[winsig.c:597]
A1B903    mysqld-debug.exe!abort()[abort.c:78]
A150F4    mysqld-debug.exe!_wassert()[assert.c:212]
432ED8    mysqld-debug.exe!Field_string::store()[field.cc:5920]
4A9E55    mysqld-debug.exe!Item::save_in_field()[item.cc:4696]
651264    mysqld-debug.exe!fill_record()[sql_base.cc:5815]
650FBC    mysqld-debug.exe!fill_record_n_invoke_before_triggers()[sql_base.cc:5860]
68E034    mysqld-debug.exe!mysql_insert()[sql_insert.cc:764]
6B44F4    mysqld-debug.exe!mysql_execute_command()[sql_parse.cc:3666]
6BD040    mysqld-debug.exe!mysql_parse()[sql_parse.cc:6267]
573F16    mysqld-debug.exe!Query_log_event::exec_event()[log_event.cc:2044]
5737F9    mysqld-debug.exe!Query_log_event::exec_event()[log_event.cc:1896]
5EE408    mysqld-debug.exe!exec_relay_log_event()[slave.cc:3420]
5ED695    mysqld-debug.exe!handle_slave_sql()[slave.cc:4030]
869055    mysqld-debug.exe!pthread_start()[my_winthread.c:85]
A2C625    mysqld-debug.exe!_callthreadstart()[thread.c:295]
A2C5F7    mysqld-debug.exe!_threadstart()[thread.c:277]
D6B69A    kernel32.dll!BaseThreadStart()
[7 Apr 2009 10:47] Shane Bester
testcase caused these errors on valgrind for 5.1.32 (Invalid read of size ...)

Attachment: bug41725_5.1.32_slave_valgrind_errors.txt (text/plain), 16.44 KiB.

[7 Apr 2009 10:55] Shane Bester
running mysqld-debug on same 32-bit linux box as above, under valgrind, caused the process to crash (in addition to prior valgrind warnings):

mysqld-debug: field.cc:6383: virtual int Field_string::store(const char*, uint, CHARSET_INFO*): Assertion `table->in_use == _current_thd()' failed.
090407  6:55:54 - mysqld got signal 6 ;
[9 Apr 2009 0:17] Bugs System
A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/71706

2748 Luis Soares	2009-04-09
      BUG#41725: slave crashes when inserting into temporary table after
      stop/start slave
      
      When the slave is replicating many temporary tables and inserts into
      those, running 'stop slave' then 'start slave' can crash the
      server. This seems to be caused due to the fact temporary tables are
      saved between slave threads restart but, regardless of the restart,
      table->in_use still references old thread instead of new one.
      
      This patch addresses this issue by resetting the reference to the new
      slave thread on slave thread restart.
[11 Apr 2009 16:12] Bugs System
A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/71889

2748 Luis Soares	2009-04-11
      BUG#41725: slave crashes when inserting into temporary table after
      stop/start slave
      
      When stopping and restarting the slave while it is replicating
      temporary tables, the server would crash or raise an assertion
      failure. This was due to the fact that although temporary tables are
      saved between slave threads restart, the reference to the thread in
      use (table->in_use) was not being properly updated when the restart
      happened (it would still reference the old/invalid thread instead of
      the new one).
      
      This patch addresses this issue by resetting the reference to the new
      slave thread on slave thread restart.
     @ mysql-test/r/rpl_temporary.result
        Result file.
     @ mysql-test/t/rpl_temporary.test
        Test case that checks that both failures go away.
     @ sql/slave.cc
        Changed slave.cc to reset sql_thd reference in temporary tables.
[23 May 2009 0:59] Luis Soares
Pushed to 5.0-bugteam, 5.1-bugteam, 6.0-bugteam.
[28 May 2009 7:42] Bugs System
Pushed into 5.0.83 (revid:joro@sun.com-20090528073529-q9b8s60vlpu28fny) (version source revid:luis.soares@sun.com-20090522231521-47kktnuct6jp1x07) (merge vers: 5.0.83) (pib:6)
[28 May 2009 8:15] Bugs System
Pushed into 5.1.36 (revid:joro@sun.com-20090528073639-yohsb4q1jzg7ycws) (version source revid:luis.soares@sun.com-20090522232941-b4wtxiallbgqscx6) (merge vers: 5.1.36) (pib:6)
[28 May 2009 12:56] Jon Stephens
Documented bugfix in the 5.0.83 and 5.1.36 changelogs, as follows:

    When stopping and restarting the slave while it was replicating 
    temporary tables, the slave server could crash or raise an assertion 
    failure. This was due to the fact that, although temporary tables were 
    saved between slave thread restarts, the reference to the thread being 
    used (table->in_use) was not being properly updated when restarting, 
    continuing to reference the old thread instead of the new one. 

    This  issue affected statement-based replication only.
[28 May 2009 13:08] Jon Stephens
Set to NDI pending push to 5.1+ trees.
[17 Jun 2009 19:23] Bugs System
Pushed into 5.4.4-alpha (revid:alik@sun.com-20090616183122-chjzbaa30qopdra9) (version source revid:luis.soares@sun.com-20090522234408-i3pxs02zbza7aova) (merge vers: 6.0.12-alpha) (pib:11)
[24 Jun 2009 14:17] Jon Stephens
Documented bugfix in the 5.4.4 changelog as follows:

        When replicating many temporary tables and inserts into those
        tables, executing STOP SLAVE followed by START SLAVE could cause
        the server to crash. This appeared to be caused by temporary
        tables preserving outdated thread references after the restart.
[12 Aug 2009 22:08] Paul Dubois
Noted in 5.4.2 changelog because next 5.4 version will be 5.4.2 and not 5.4.4.
[14 Aug 2009 22:57] Paul Dubois
Ignore previous comment about 5.4.2.
[26 Aug 2009 13:46] Bugs System
Pushed into 5.1.37-ndb-7.0.8 (revid:jonas@mysql.com-20090826132541-yablppc59e3yb54l) (version source revid:jonas@mysql.com-20090826132541-yablppc59e3yb54l) (merge vers: 5.1.37-ndb-7.0.8) (pib:11)
[26 Aug 2009 13:46] Bugs System
Pushed into 5.1.37-ndb-6.3.27 (revid:jonas@mysql.com-20090826105955-bkj027t47gfbamnc) (version source revid:jonas@mysql.com-20090826105955-bkj027t47gfbamnc) (merge vers: 5.1.37-ndb-6.3.27) (pib:11)
[26 Aug 2009 13:48] Bugs System
Pushed into 5.1.37-ndb-6.2.19 (revid:jonas@mysql.com-20090825194404-37rtosk049t9koc4) (version source revid:jonas@mysql.com-20090825194404-37rtosk049t9koc4) (merge vers: 5.1.37-ndb-6.2.19) (pib:11)
[27 Aug 2009 16:32] Bugs System
Pushed into 5.1.35-ndb-7.1.0 (revid:magnus.blaudd@sun.com-20090827163030-6o3kk6r2oua159hr) (version source revid:jonas@mysql.com-20090826132541-yablppc59e3yb54l) (merge vers: 5.1.37-ndb-7.0.8) (pib:11)
[7 Oct 2009 14:37] Paul Dubois
The 5.4 fix has been pushed to 5.4.2.