Bug #38594 Falcon crash in MemMgr and Sync object during exit of mysqld
Submitted: 6 Aug 2008 8:36 Modified: 8 Jan 2009 10:31
Reporter: Olav Sandstå Email Updates:
Status: Closed Impact on me:
None 
Category:MySQL Server: Falcon storage engine Severity:S3 (Non-critical)
Version:6.0.6 OS:Solaris (Solaris 10 on AMD Opteron)
Assigned to: Vladislav Vaintroub CPU Architecture:Any
Tags: Sun Studio
Triage: D1 (Critical)

[6 Aug 2008 8:36] Olav Sandstå
Description:
When running mysql_install_db I observe the following crash of mysqld when having compiled with --with-debug using the Sun Studio 12 compiler:

Output from the process:
========================

Installing MySQL system tables...
Pure virtual function called
safe_mutex: Trying to lock unitialized mutex at my_getsystime.c, line 171
Abort - core dumped

Installation of system tables failed!  Examine the logs in
/export/home/tmp/mysqldb for more information.

Call stack from core file:
==========================

t@1 (l@1) terminated by signal ABRT (Abort)
0xfec24aa7: __lwp_kill+0x0007:  jae      __lwp_kill+0x15        [ 0xfec24ab5, .+0xe ]
Current function is safe_mutex_lock
  107       abort();
(dbx) where -h
current thread: t@1
  [1] __lwp_kill(0x1, 0x6), at 0xfec24aa7 
  [2] _thr_kill(0x1, 0x6), at 0xfec22250 
  [3] raise(0x6), at 0xfebd1217 
  [4] abort(0xfef02a00, 0xfec4d000, 0xfec24670, 0x8c11e59, 0x2, 0x0), at 0xfebb1629 
=>[5] safe_mutex_lock(mp = 0x8e9ad84, try_lock = '\0', file = 0x8c0d7cc "my_getsystime.c", line = 171U), line 107 in "thr_mutex.c"
  [6] my_micro_time_and_time(time_arg = 0x8046d44), line 171 in "my_getsystime.c"
  [7] my_time(flags = 0), line 74 in "my_getsystime.c"
  [8] handle_segfault(sig = 6), line 2534 in "mysqld.cc"
  [9] __sighndlr(0x6, 0x0, 0x8046e44, 0x83765b0), at 0xfec23e6f 
  [10] call_user_handler(0x6, 0x0, 0x8046e44), at 0xfec1a30e 
  [11] sigacthandler(0x6, 0x0, 0x8046e44, 0xf, 0x0, 0x807007), at 0xfec1a48e 
  ---- called from signal handler with signal 6 (SIGABRT) ------
  [12] __lwp_kill(0x1, 0x6), at 0xfec24aa7 
  [13] raise(0x6), at 0xfebd1217 
  [14] abort(0xfed9ff44, 0x80470e8, 0xfeffdcc8, 0x80470e8, 0x8617e20, 0x8d29750), at 0xfebb15c0 
  [15] __Crun::pure_error(0x8d29750, 0x8047164, 0x1, 0x0, 0x0, 0x0), at 0xfec6656a 
  [16] Sync::lock(this = 0x8047164, type = Exclusive), line 58 in "Sync.cpp"
  [17] MemMgr::alloc(this = 0x8d296c0, length = 48), line 328 in "MemMgr.cpp"
  [18] MemMgr::allocateDebug(this = 0x8d296c0, size = 24, fileName = 0x8b3ae34 "MemoryManager.h", line = 48), line 539 in "MemMgr.cpp"
  [19] MemMgrAllocateDebug(s = 24U, file = 0x8b3ae34 "MemoryManager.h", line = 48), line 124 in "MemMgr.cpp"
  [20] operator new(s = 24U), line 48 in "MemoryManager.h"
  [21] std::basic_filebuf<wchar_t,std::char_traits<wchar_t> >::close(0xfede02d0), at 0xfecec0c6 
  [22] std::basic_filebuf<wchar_t,std::char_traits<wchar_t> >::~basic_filebuf(0xfede02d0), at 0xfece8917 
  [23] std::__Wide_Init::~__Wide_Init(0x8047284), at 0xfed63af2 
  [24] __SUNW_fini_iostreams(0xfef001c0, 0xfec52218, 0xfed9ff44, 0xfef00200, 0xfec52218, 0xfec4d000), at 0xfed6a87b 
  [25] __Cimpl::cplus_fini(0xfec4d000, 0xfebb1d7d, 0x80474c8, 0x804734c, 0xfec4d000, 0xfec52200), at 0xfec663eb 
  [26] __cplus_fini_at_exit(0xfeffb818, 0x8374dfc, 0x0, 0x0, 0x0, 0x0), at 0xfed7c9b2 
  [27] exit(0x0, 0x0), at 0xfeba4a32 
  [28] main(argc = 7, argv = 0x8047390), line 4638 in "mysqld.cc"

Note 1: this only happens when I have compiled with --with-debug.

Note 2: The call stack contains two crashes. The initial crash in Falcon code and a second crash in the server code. The crash in the server code is reported in bug #37961 (http://bugs.mysql.com/bug.php?id=37961).

How to repeat:
1. Configure MySQL with Falcon for Sun Studio compiler with the --with-debug option.

2. Compile

3. Run mysql_install_db
[6 Aug 2008 9:59] Olav Sandstå
To verify whether the MemMgr was a valid object or had already been destructed at the time of the crash I added a fprintf to the constructor and destructor of MemMgr and an assert on a status variable for whether or not the destructor had been run on it. The result:

Installing MySQL system tables...
MemMgr::Mgr(int,int,int)
MemMgr::Mgr(int,int,int)
MemMgr::~MemMgr
MemMgr::~MemMgr
Assertion failed: init == 6, file MemMgr.cpp, line 340

So the cause for this crash is basically that we are trying to allocate a new object from an already destructed Memory Manager (which also was Vlad's theory).
[6 Aug 2008 12:25] Vladislav Vaintroub
Fix is in http://lists.mysql.com/commits/50993
Pushed to falcon-team
[7 Aug 2008 18:21] Bugs System
A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/51134

2770 Vladislav Vaintroub	2008-08-07 [merge]
      merge
[7 Aug 2008 18:21] Bugs System
A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/51135

2770 Vladislav Vaintroub	2008-08-07 [merge]
      merge
[22 Aug 2008 22:07] Kevin Lewis
This fix is in version 6.0.7
[28 Aug 2008 20:17] Bugs System
Pushed into 6.0.7-alpha  (revid:cbell@mysql.com-20080822132131-uveo6wiuecy6m2b8) (version source revid:cbell@mysql.com-20080822132131-uveo6wiuecy6m2b8) (pib:3)
[29 Aug 2008 0:06] Kevin Lewis
This solution is temporary. The real problem is outlined in Bug#38598.  Once we can use the Falcon memory manager without overriding global new and delete, then the Falcon Memory Manager should be reverted back to only using itself.  It should be used within Falcon only when it is instantiated.  This patch makes it switch automatically to use plain malloc() & free() if the memory manager is not alive.  This solution has caused follow-on problems when a Jstring was allocated before the MemMgr was alive, but freed after.  See the latest patch for Bug#38041; http://lists.mysql.com/commits/52288.
[14 Sep 2008 5:53] Bugs System
Pushed into 6.0.7-alpha  (revid:vvaintroub@mysql.com-20080806115321-60zhg0yehaxhca7c) (version source revid:john.embretsen@sun.com-20080724122511-9c0oudz1xrdrs6y6) (pib:3)
[8 Jan 2009 10:31] MC Brown
A note has been added to the 6.0.7 changelog: 

When using mysql_install_db on MySQL built with Sun Studio 12 with the --with-debug option enabled, the server would crash