Bug #29431 killing an insert delayed thread causes crash
Submitted: 29 Jun 2007 0:01 Modified: 27 Jul 2007 7:09
Reporter: Shane Bester
Status: Closed
Category:Server: Locking Severity:S3 (Non-critical)
Version:4.1,5.0.46,5.1 OS:Any
Assigned to: Konstantin Osipov Target Version:
Tags: hang, crash, insert delayed, stack overrun

[29 Jun 2007 0:01] Shane Bester 
Description:
seen mysqld process die after a thread sent KILL <id> to the server and <id> happened to
be some insert delayed query running.

mostly there's no stack trace in the logs because it appears mysqld was killed by
kernel.

once, I saw some out of memory messages.

How to repeat:
will make a testcase later.

Suggested fix:
.
[29 Jun 2007 0:52] Shane Bester 
there's a massive stack overrun happening in this crash. even debug build in debugger
didn't catch a stack trace.
[29 Jun 2007 0:54] Shane Bester 
testcase. see top of fille for user, host, gcc compile example

Attachment: bug29431.c (text/plain), 8.08 KiB.
[29 Jun 2007 0:57] Shane Bester 
some testcase output, and error log

Attachment: bug29431_testcase_output.txt (text/plain), 4.82 KiB.
[29 Jun 2007 1:10] Shane Bester 
some notes.  i ran debug server which crashed much quicker than release server. 
5.0.36-enterprise-gpl-glib23 did crash, after a few minutes.  debug 5.0.46 crashes much
quicker.

run the server to allow fast connections:

mysqld_safe --skip-grant-tables --skip-name-resolve --skip-innodb --back_log=10000
[29 Jun 2007 1:23] Shane Bester 
i couldn't crash 5.1.20, but all threads got hanged

Attachment: bug29431_hang_on_5.1.20.txt (text/plain), 4.58 KiB.
[18 Jul 2007 1:01] Konstantin Osipov 
Also repeatable in 4.1
[19 Jul 2007 1:13] Bugs System 
A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/31144

ChangeSet@1.2533, 2007-07-19 03:13:10+04:00, kostja@bodhi.(none) +1 -0
  A fix for Bug#29431 killing an insert delayed thread causes crash
  No test case, since the bug requires a stress case with 30 INSERT DELAYED
  threads and 1 killer thread to repeat. The patch is verified
  manually.
  
  The server that is running DELAYED inserts would deadlock itself
  or crash under high load if some of the delayed threads were KILLed
  in the meanwhile.
  
  The fix is to change internal lock acquisition order of delayed inserts
  subsystem and to ensure that
  Delayed_insert::table_list::db does not point to volatile memory in some 
  cases.
  For details, please see a comment for sql_insert.cc.
[19 Jul 2007 1:28] Bugs System 
A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/31145

ChangeSet@1.2533, 2007-07-19 03:28:47+04:00, kostja@bodhi.(none) +1 -0
  A fix for Bug#29431 killing an insert delayed thread causes crash
  No test case, since the bug requires a stress case with 30 INSERT DELAYED
  threads and 1 killer thread to repeat. The patch is verified
  manually.
  
  The server that is running DELAYED inserts would deadlock itself
  or crash under high load if some of the delayed threads were KILLed
  in the meanwhile.
  
  The fix is to change internal lock acquisition order of delayed inserts
  subsystem and to ensure that
  Delayed_insert::table_list::db does not point to volatile memory in some 
  cases.
  For details, please see a comment for sql_insert.cc.
[19 Jul 2007 17:29] Bugs System 
A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/31175

ChangeSet@1.2533, 2007-07-19 19:28:00+04:00, kostja@bodhi.(none) +1 -0
  A fix for Bug#29431 killing an insert delayed thread causes crash
  No test case, since the bug requires a stress case with 30 INSERT DELAYED
  threads and 1 killer thread to repeat. The patch is verified
  manually.
  Review fixes.
  
  The server that is running DELAYED inserts would deadlock itself
  or crash under high load if some of the delayed threads were KILLed
  in the meanwhile.
  
  The fix is to change internal lock acquisition order of delayed inserts
  subsystem and to ensure that
  Delayed_insert::table_list::db does not point to volatile memory in some 
  cases.
  For details, please see a comment for sql_insert.cc.
[19 Jul 2007 17:40] Konstantin Osipov 
Queued into 5.0-runtime
[19 Jul 2007 19:10] Bugs System 
A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/31184

ChangeSet@1.2559, 2007-07-19 21:10:19+04:00, kostja@bodhi.(none) +1 -0
  A follow-up fix for Bug#29431 "killing an insert delayed thread causes 
  crash" in 5.1
[25 Jul 2007 10:47] Bugs System 
Pushed into 5.0.48
[25 Jul 2007 10:48] Bugs System 
Pushed into 5.1.21-beta
[27 Jul 2007 7:09] Paul DuBois 
Noted in 5.0.48, 5.1.21 changelogs.

Killing an INSERT DELAYED thread caused a server crash.