Bug #55442 mysqld debug crashes while running myisam_crash_before_flush_keys.test
Submitted: 21 Jul 2010 14:21 Modified: 24 Mar 2011 22:04
Reporter: Georgi Kodinov Email Updates:
Status: Closed Impact on me:
None 
Category:Tools: MTR / mysql-test-run Severity:S3 (Non-critical)
Version:5.1-bugteam-debug OS:Mac OS X (10.6.4 SnowLeopard)
Assigned to: Bjørn Munch CPU Architecture:Any

[21 Jul 2010 14:21] Georgi Kodinov
Description:
I get a lot of "mysqld quit unexpectedly" popups on my mac while running the binlog test suite. 
It contains mostly this callstack : 

Thread 2 Crashed:
0   libSystem.B.dylib             	0x00007fff84172b6e __semwait_signal_nocancel + 10
1   libSystem.B.dylib             	0x00007fff84172a70 nanosleep$NOCANCEL + 129
2   libSystem.B.dylib             	0x00007fff841cf3c6 usleep$NOCANCEL + 57
3   libSystem.B.dylib             	0x00007fff841ee91d __abort + 113
4   libSystem.B.dylib             	0x00007fff841ee981 abort_report_np + 0
5   mysqld                        	0x00000001001f5475 MYSQL_BIN_LOG::open(char const*, enum_log_type, char const*, cache_type, bool, unsigned long, bool, bool) + 551 (log.cc:2589)
6   mysqld                        	0x00000001001f6597 MYSQL_BIN_LOG::new_file_impl(bool) + 1263 (log.cc:3828)
7   mysqld                        	0x00000001001f6668 MYSQL_BIN_LOG::new_file_without_locking() + 26 (log.cc:3725)
8   mysqld                        	0x00000001001f6700 MYSQL_BIN_LOG::rotate_and_purge(unsigned int) + 150 (log.cc:4529)
9   mysqld                        	0x0000000100122c09 reload_acl_and_cache(THD*, unsigned long, TABLE_LIST*, bool*) + 508 (sql_parse.cc:6903)
10  mysqld                        	0x000000010012fc0f mysql_execute_command(THD*) + 26186 (sql_parse.cc:4026)
11  mysqld                        	0x00000001001333d1 mysql_parse(THD*, char const*, unsigned int, char const**) + 727 (sql_parse.cc:6022)
12  mysqld                        	0x000000010013434d dispatch_command(enum_server_command, THD*, char*, unsigned int) + 3504 (sql_parse.cc:1262)
13  mysqld                        	0x00000001001359a1 do_command(THD*) + 644 (sql_parse.cc:888)
14  mysqld                        	0x00000001001208fc handle_one_connection + 342 (sql_connect.cc:1136)
15  libSystem.B.dylib             	0x00007fff84139456 _pthread_start + 331
16  libSystem.B.dylib             	0x00007fff84139309 thread_start + 13

How to repeat:
compile with BUILD/compile-pentium-debug-max on a 64 bit MacOSX 10.6 and run /mysql-test-run.pl --force --mem --parallel=2 --skip-ndb --skip-rpl --retry=0 --max-test-fail=0 --suite=main,binlog,funcs_1

Suggested fix:
Don't crash ?
[21 Jul 2010 15:48] Valeriy Kravchuk
I was not able to repeat this with current mysql-5.1 from bzr on Mac OS X 10.5.x:

...
binlog.binlog_killed 'stmt'              w1 [ pass ]   3231
binlog.binlog_row_mysqlbinlog_db_filter 'row' w1 [ pass ]   3894
binlog.binlog_tbl_metadata 'row'         w1 [ pass ]   2286
binlog.binlog_row_failure_mixing_engines 'row' w2 [ pass ]   2868
------------------------------------------------------------
The servers were restarted 50 times
Spent 631.288 of 447 seconds executing testcases

Completed: All 74 tests were successful.
[22 Jul 2010 8:56] Georgi Kodinov
Seems to be version specific. Dunno if it's the gcc or the os version.
Here's what I have : 
$ sw_vers 
ProductName:    Mac OS X
ProductVersion: 10.6.4
BuildVersion:   10F569
$ gcc --version
i686-apple-darwin10-gcc-4.2.1 (GCC) 4.2.1 (Apple Inc. build 5659)
Copyright (C) 2007 Free Software Foundation, Inc.

Xcode version 3.2.2 64 bit.
[26 Dec 2010 15:57] Valeriy Kravchuk
As thins is OS-specific and we do not have access the same OS version at the moment, please, check if the problem is still repeatable with a recent version of mysql-5.1 (5.1.55)
[3 Jan 2011 13:43] Georgi Kodinov
I still get this, but with much less frequency (1 for the whole test suite run) using the latest 5.1-bugteam tree.
Here's the updated call-stack:
Thread 2 Crashed:
0   libSystem.B.dylib             	0x00007fff8905de4e __semwait_signal_nocancel + 10
1   libSystem.B.dylib             	0x00007fff8905dd50 nanosleep$NOCANCEL + 129
2   libSystem.B.dylib             	0x00007fff890ba6a2 usleep$NOCANCEL + 57
3   libSystem.B.dylib             	0x00007fff890d9cd4 abort + 93
4   mysqld                        	0x000000010045c49d mi_close + 670
5   mysqld                        	0x0000000100480699 ha_myisam::close() + 51
6   mysqld                        	0x000000010018d3a3 closefrm(st_table*, bool) + 166 (table.cc:1997)
7   mysqld                        	0x0000000100178743 intern_close_table(st_table*) + 295 (sql_base.cc:783)
8   mysqld                        	0x00000001001787f0 free_cache_entry(st_table*) + 131 (sql_base.cc:805)
9   mysqld                        	0x00000001004b785c my_hash_delete + 1162
10  mysqld                        	0x00000001001761d0 remove_table_from_cache(THD*, char const*, char const*, unsigned int) + 1239 (sql_base.cc:8592)
11  mysqld                        	0x000000010017b8b0 close_cached_tables(THD*, TABLE_LIST*, bool, bool, bool) + 554 (sql_base.cc:932)
12  mysqld                        	0x000000010011dd65 reload_acl_and_cache(THD*, unsigned long, TABLE_LIST*, int*) + 1088 (sql_parse.cc:7034)
13  mysqld                        	0x000000010012ad14 mysql_execute_command(THD*) + 26639 (sql_parse.cc:4066)
14  mysqld                        	0x000000010012e4d8 mysql_parse(THD*, char*, unsigned int, char const**) + 727 (sql_parse.cc:6075)
15  mysqld                        	0x000000010012f451 dispatch_command(enum_server_command, THD*, char*, unsigned int) + 3501 (sql_parse.cc:1263)
16  mysqld                        	0x0000000100130aa5 do_command(THD*) + 644 (sql_parse.cc:889)
17  mysqld                        	0x000000010011b813 handle_one_connection + 342 (sql_connect.cc:1149)
18  libSystem.B.dylib             	0x00007fff89024536 _pthread_start + 331
19  libSystem.B.dylib             	0x00007fff890243e9 thread_start + 13
[4 Jan 2011 12:04] Jon Olav Hauglid
I'm not able to repeat this crash with the mysql-5.1 tree (revno: 3531).

$ sw_vers 
ProductName:	Mac OS X
ProductVersion:	10.6.5
BuildVersion:	10H574

$ gcc --version
i686-apple-darwin10-gcc-4.2.1 (GCC) 4.2.1 (Apple Inc. build 5664)

XCode 3.2.5

I've tried both 32 and 64 bit.
[4 Jan 2011 12:45] Georgi Kodinov
Jon Olav, it's not a crash per se : the test goes on without a hint of a failing test. All I get is the OSX system crash dialog with the callstack above.
[4 Jan 2011 12:51] Jon Olav Hauglid
I did not get any OSX system popups about mysqld crashes either.
[4 Jan 2011 12:54] Georgi Kodinov
You may want to check in the CrashReporterPrefs app if crash reporter is enabled : http://en.wikipedia.org/wiki/Crash_Reporter_(Mac_OS_X)
[4 Jan 2011 13:22] Jon Olav Hauglid
Missed that the main suite was included in the test run.

Able to repeat it now, but not with the binlog suite. Instead I get it consistently when running main.myisam_crash_before_flush_keys.test
[4 Jan 2011 14:08] Georgi Kodinov
probably related to bug #41330.
[13 Jan 2011 10:37] Bjørn Munch
It looks to me like you simply want to disable the Crash reporter. Is there a way for mtr to do that? Or for mtr to make mysqld itself do it?

This is not related to whether the crash was expected by mtr or not. Some test will intentionally crash the server, so you really want to disable Crash reporter.

I am not able to test this myself, not having physical access to a Mac, unless I can borrow one.
[13 Jan 2011 11:52] Bjørn Munch
So, should the Crash Reporter be disabled for all mtr runs, so it also doesn't trigger when there's a real crash, or only for those tests that intentionally crash the server? Not that mtr can detect this; each such test would have to be identified and changed.

This is not a promise that mtr/tests can actually do this... Alternative is to close as !Bug and document that you may want to turn the CrashReporter off before running mtr tests.
[14 Jan 2011 13:39] Bugs System
A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/128755

2969 Bjorn Munch	2011-01-14
      Bug #55442 mysqld debug crashes while running myisam_crash_before_flush_keys.test
      This will cause affected tests to skip if CrashReporter would popup
      Found 4 tests that needed modification
[18 Jan 2011 11:12] Georgi Kodinov
Bjorn,

Are you aware of any tests that actually cause the server to crash and expect this ? 
I'm not.
So while I appreciate the neat way you've detected whether the crash reporter is present it's not the right way to go.
IMHO this was classified as a mtr bug because the actual mtr output doesn't show the crash whereas crashreporter does. So there's either something wrong with the test or the way mtr detects crashes.
[18 Jan 2011 11:21] Bjørn Munch
Yes, the 4 tests that I fixed in my patch all intentionally crash the server and would trigger CrashReporter.

There is no mtr bug; there crashes are expected and therefore should *not* be reported in any way. I could have tried to turn the CrashReporter off, then on again while mtr is running, but I don't like the idea of fiddling with user settings that way. Besides, that would also prevent CrashReporter for unexpected crashes, and I presume the user might actually want to have that enabled.
[26 Jan 2011 11:47] Bjørn Munch
The patch suggested is IMHO the best mtr can do, but I'd like some feedback as to whether this is wanted or not. I have identified 4 tests that needed to be skipped, I don't think there are more (currently).
[27 Jan 2011 14:13] Bjørn Munch
There may be an alternative. I was pointed to include/my_dbug.h which says:

#ifdef __WIN__
#define DBUG_SUICIDE() DBUG_ABORT()
#else
....

And DBUG_ABORT() ends with _exit() instead of abort(). If this is done in Mac OS X too, it might not trigger the CrashReporter? If this idea is pursued it will be a server issue, not MTR. Not having access to a Mac I cannot easily test it myself.
[10 Feb 2011 12:56] Bjørn Munch
I'd like some feedback on my last comment. Is this a viable alternative, or should I just proceed with the proposed mtr fix?
[24 Mar 2011 22:04] Paul DuBois
Changes to test suite. No changelog entry needed.

CHANGESET - http://lists.mysql.com/commits/133046