| Bug #40938 | mysqladmin shutdown hangs on HP/UX | ||
|---|---|---|---|
| Submitted: | 21 Nov 2008 18:42 | Modified: | 2 Dec 2010 17:43 |
| Reporter: | Scott Noyes | Email Updates: | |
| Status: | Not a Bug | Impact on me: | |
| Category: | MySQL Server: InnoDB storage engine | Severity: | S2 (Serious) |
| Version: | 5.0.75 | OS: | HP/UX (11.31) |
| Assigned to: | Kent Boortz | CPU Architecture: | Any |
[21 Nov 2008 18:42]
Scott Noyes
[22 Nov 2008 8:00]
Sveta Smirnova
Thank you for the report. I can not repeat described behavior with current version 5.0.70. Please consider to upgrade.
[3 Dec 2008 16:06]
Valeriy Kravchuk
Re-opened as the bug is platform-specific it seems, HP-UX 11.31 on Itanium.
[27 Jan 2009 22:16]
Sveta Smirnova
Hang happens only if modify a table during session. Like: create table t1(f1 int); insert into t1 values(1),(2); update t1 set f1=3 where f1=1; If remove update hang does not happen.
[28 Jan 2009 21:39]
Sveta Smirnova
Did additional testing.
Server hangs in innobase_shutdown_for_mysql:
HP gdb 5.9 for HP Itanium (32 or 64 bit) and target HP-UX 11.2x.
..
Attaching to program: /PATH/./libexec/mysqld, process 7439
warning: Load module /usr/local/lib/libstdc++.so has been stripped.
Debugging information is not available.
0x60000000c03ecf10:0 in __ksleep+0x30 () from /usr/lib/hpux32/libc.so.1
(gdb) infor threads
Undefined command: "infor". Try "help".
(gdb) info threads
14 system thread 3291912 Priority:154 0x60000000c03ecf10:0 in __ksleep
+0x30 () from /usr/lib/hpux32/libc.so.1
10 system thread 3291530 Priority:168 0x60000000c03f5d30:0 in __sigwait_sys+0x30 () from /usr/lib/hpux32/libc.so.1
* 1 system thread 3291520 Priority:137 0x60000000c03ecf10:0 in __ksleep
+0x30 () from /usr/lib/hpux32/libc.so.1
(gdb) thread 14
[Switching to thread 14 (system thread 3291912)]
#0 0x60000000c03ecf10:0 in __ksleep+0x30 () from /usr/lib/hpux32/libc.so.1
(gdb) bt full
#0 0x60000000c03ecf10:0 in __ksleep+0x30 () from /usr/lib/hpux32/libc.so.1
No symbol table info available.
#1 0x60000000c01804d0:0 in __mxn_sleep+0xc70 ()
from /usr/lib/hpux32/libpthread.so.1
No symbol table info available.
#2 0x60000000c01009e0:0 in <unknown_procedure> + 0x4f0 ()
from /usr/lib/hpux32/libpthread.so.1
No symbol table info available.
#3 0x60000000c0105f30:0 in pthread_mutex_lock+0x270 ()
from /usr/lib%2
[28 Jan 2009 21:45]
Sveta Smirnova
What also I did.
1. Use 5.0.75 source tar.gz
2. ./configure --with-debug=full
3. See previous comments under which circumstances hang is repeatable. Additionally customer uses SQLyog to modify table. Probably is sends additional statements like SHOW TABLE STATUS, because simple using of mysql command line client does not lead to such results.
4. Connect to mysql via SQLyog, modify table, issue mysqladmin shutdown.
5. Wait forever
6. I did following change:
$diff -u innobase/srv/srv0start.c innobase/srv/srv0start.c.new
--- innobase/srv/srv0start.c 2008-12-18 18:19:37.000000000 +0100
+++ innobase/srv/srv0start.c.new 2009-01-28 22:42:51.000000000 +0100
@@ -1836,10 +1836,12 @@
mem_free(srv_monitor_file_name);
}
}
+/*
if (srv_dict_tmpfile) {
fclose(srv_dict_tmpfile);
srv_dict_tmpfile = 0;
}
+*/
if (srv_misc_tmpfile) {
fclose(srv_misc_tmpfile);
7. Recompiled MySQL
8. Repeat step #4.
9. No hang, mysqld shutdowns successfully.
Additionally converted InnoDB table to MyISAM, no hang either.
So I verify this report as InnoDB bug. Bug is repeatable only on this specific platform.
[29 Jan 2009 15:59]
Mikhail Izioumtchenko
from the stack supplied in the bug it's not evident that the shutdown routine, or fclose(), as it would follow from the investigation above. I don't doubt either point but could you still provide output of 'thr a a bt' in gdb attached to the hanging process. However if we do see fclose() hanging this wouldn't be InnoDB but rather OS problem. Knowing the name of that file and what other processes or threads have it open could still be of interest.
[30 Jan 2009 7:55]
Sveta Smirnova
Michael, bt full of the thread attached.
[29 Apr 2009 19:53]
Sveta Smirnova
Michael, can you answer my last question in the private comment? Thanks in advance.
[24 Jun 2009 17:07]
Matthew Lord
It looks like the cause is the first critical bug listed here as part of the PHCO_38050 patch set: http://www13.itrc.hp.com/service/patch/patchDetail.do?BC=main|patchDetail{PHCO_38050,{hpux... Based on the debugging that I've done with gdb I can see that the hangs are occurring in the pthread library when InnoDB is doing an ABORT which fits perfectly with HP's description of the bug: PHCO_38050: ABORT HANG Pthread cancel signal is getting consumed by other routine. This will prevent the delivery of the signal to the target thread and the target thread does not exit, subsequently the application hangs. Multithreaded applications that access TLS data types that are 16 bytes in size sometimes get alignment trap. Please also see: http://bugs.mysql.com/bug.php?id=45691
[24 Jun 2009 17:14]
Matthew Lord
I would recommend that ALL of the following patchsets from HP be installed: PHKL_37465 1.0 thread level enhancements patch PHKL_37457 1.0 thread level signal enhancements patch PHCO_38050 1.0 pthread library cumulative patch PHCO_38048 1.0 libc cumulative patch PHCO_38044 1.0 libc cumulative header file patch PHCO_37128 1.0 libc manpage cumulative patch
[25 Jun 2009 6:49]
Marko Mäkelä
This could be fixed in InnoDB code by defining UT_DBG_PANIC in ut0dbg.h as abort() and UT_DBG_STOP as a null operation. MySQL 5.1 does that, but only when compiled with GCC. This works if abort() is guaranteed to terminate all threads of the process. If I remember correctly, on Windows, abort() won't terminate all threads, or it will not produce a useful call trace. Can someone who has access to the hardware please test this?
[2 Dec 2010 17:44]
Sveta Smirnova
Latest OS patches solved the problem, so this is not MySQL bug.
