Bug #61454 MySQL debug build locking up, SIGABRT
Submitted: 8 Jun 2011 21:56 Modified: 2 Sep 2011 17:24
Reporter: Chris Wagner Email Updates:
Status: No Feedback Impact on me:
None 
Category:MySQL Server: General Severity:S6 (Debug Builds)
Version:5.5.13-debug OS:Linux (Ubuntu)
Assigned to: CPU Architecture:Any
Tags: aborted, debug, lock up, locked up, locking up, sigabrt, signal 6

[8 Jun 2011 21:56] Chris Wagner
Description:
While trying to debug some things I used the mysqld-debug executable that comes with the the 5.5.13 distrubution.  It seems to run OK for a while but then at some point locks up.  I can tell when it happens because as I monitor it in MySQL Workbench I can see connections build up.  WB tries to run show global status every few seconds but even this simple statement locks up indefinately when this happens.  That causes WB to try to reconnect and run the show statement again.  This causes thread after thread to build up until the max connections is reached.  It's not WB's fault because it does the same thing from the mysql command line.  Running show global status just sits there forever.  Watching the process in top, I can see that mysqld-debug has stopped doing any real work.  Other scripts which are using the database also freeze.

I was able to use mysqladmin to make the process show some status information while it was in the locked up state.  That is in the attached file.  Eventually I have to kill the process with kill -9 because the normal shutdown process never completes.  Even kill -TERM and kill -INT do nothing.

OS info:
Linux version 2.6.35-25-server (buildd@allspice) (gcc version 4.4.3 (Ubuntu 4.4.
3-4ubuntu5) ) #44~lucid1-Ubuntu SMP Tue Jan 25 19:34:09 UTC 2011
Ubuntu 2.6.35-25.44~lucid1-server 2.6.35.10

Thanks.

How to repeat:
Run mysqld-debug, wait a while.
[8 Jun 2011 21:57] Chris Wagner
status information and error log output while daemon is locked up.

Attachment: lockedup1.txt (text/plain), 62.13 KiB.

[8 Jun 2011 22:01] Chris Wagner
I should add that mysqladmin extended-status does not work while the daemon is in this state.
[8 Jun 2011 22:02] Chris Wagner
I should add that mysqladmin extended-status does not work while the daemon is in this state.
[14 Jul 2011 18:54] Sveta Smirnova
Thank you for the report.

Please try with new version 5.5.14 and if problem still exist try to run query from the log to be sure if this query causes the problem.

INSERT IGNORE INTO `profiler`.`sourcenetrep` SELECT '13939', `sid`, `src_ip` & 4294967295 AS `/24 subnet`, 4294967295, 0, sum(`src_bytes`), sum(`dst_bytes`), sum(`src_pkts` + `dst_pkts`), count(*) AS `count` FROM `profiler`.`resultset9` GROUP BY `sid`, `/24 subnet`
[14 Jul 2011 18:59] MySQL Verification Team
why do you use a debug build ?  the problem here is a bug that happens after somebody kills a connection.  check the error log, it says KILL_CONNECTION
[18 Jul 2011 21:36] Chris Wagner
The query from the log definately isn't the problem.  That's just what happened to be running at the time.  Are u aware of some change in .14 that could affect this?  "Try the next version" isn't a troubleshooting procedure.  We've avoided the bug with the normal version and we can't play with our production system to verify that "it's still there."
[18 Jul 2011 23:23] Sveta Smirnova
Thank you for the feedback.

> The query from the log definately isn't the problem.  That's just what happened to be
running at the time.

So this was query which was running in time when yo decided to kill -9 server?

> Are u aware of some change in .14 that could affect this?  "Try the
next version" isn't a troubleshooting procedure. 

A lot of people including myself run latest versions of MySQL debug server for days and it does not lock up. This is something in your application. Also we don't backport bugfixes into minor  versions anyway, so there is always sense to be sure problem is not repeatable with latest one.

Regarding to error log provided I see a lot of locks, so this probably locking issue. Please send us output of SHOW PROCESSLIST and SHOW ENGINE INNODB STATUS during problem time (you should be able login as root)
[25 Jul 2011 23:57] Chris Wagner
When it locked up there was no way to do anything.  What I posted is all I could get.

It's definately not the application because it runs fine on 5.5.13 normal.  The workload wasn't even particularly demanding.  If anything it's a MySQL/Ubuntu interaction.  Regardless of any other condition, show global status should *never* fail.

There's no way for me to reload 13-debug at this point because we've gone to production.
[27 Jul 2011 19:54] Sveta Smirnova
Thank you for the feedback.

> It's definately not the application because it runs fine on 5.5.13 normal.

But your application does something what locks the server.

Please update the report when you have more information, set as "Need feedback" now.
[1 Aug 2011 22:18] Chris Wagner
Any application will have locks as part of it's normal operation.  An application lock should never cause the server to stop responding.
[2 Aug 2011 17:24] Sveta Smirnova
Chris,

I agree. But we need to know what your application does and how it lead server to such situation. Once we repeat the problem we can discuss if this is MySQL server bug or not. Since bug is confirmed we can fix it, but not having only information it locks for some reason.
[2 Sep 2011 23:00] Bugs System
No feedback was provided for this bug for over a month, so it is
being suspended automatically. If you are able to provide the
information that was originally requested, please do so and change
the status of the bug back to "Open".