MySQL Bugs: #38594: Falcon crash in MemMgr and Sync object during exit of mysqld

Bug #38594	Falcon crash in MemMgr and Sync object during exit of mysqld
Submitted:	6 Aug 2008 10:36	Modified:	8 Jan 11:31
Reporter:	Olav Sandstaa
Status:	Closed
Category:	Server: Falcon	Severity:	S3 (Non-critical)
Version:	6.0.6	OS:	Sun Solaris (Solaris 10 on AMD Opteron)
Assigned to:	Vladislav Vaintroub	Target Version:	6.0.7
Tags:	Sun Studio
Triage:	D1 (Critical)

Description:
When running mysql_install_db I observe the following crash of mysqld when having
compiled with --with-debug using the Sun Studio 12 compiler:

Output from the process:
========================

Installing MySQL system tables...
Pure virtual function called
safe_mutex: Trying to lock unitialized mutex at my_getsystime.c, line 171
Abort - core dumped

Installation of system tables failed!  Examine the logs in
/export/home/tmp/mysqldb for more information.

Call stack from core file:
==========================

t@1 (l@1) terminated by signal ABRT (Abort)
0xfec24aa7: __lwp_kill+0x0007:  jae      __lwp_kill+0x15        [ 0xfec24ab5, .+0xe ]
Current function is safe_mutex_lock
  107       abort();
(dbx) where -h
current thread: t@1
  [1] __lwp_kill(0x1, 0x6), at 0xfec24aa7 
  [2] _thr_kill(0x1, 0x6), at 0xfec22250 
  [3] raise(0x6), at 0xfebd1217 
  [4] abort(0xfef02a00, 0xfec4d000, 0xfec24670, 0x8c11e59, 0x2, 0x0), at 0xfebb1629 
=>[5] safe_mutex_lock(mp = 0x8e9ad84, try_lock = '\0', file = 0x8c0d7cc
"my_getsystime.c", line = 171U), line 107 in "thr_mutex.c"
  [6] my_micro_time_and_time(time_arg = 0x8046d44), line 171 in "my_getsystime.c"
  [7] my_time(flags = 0), line 74 in "my_getsystime.c"
  [8] handle_segfault(sig = 6), line 2534 in "mysqld.cc"
  [9] __sighndlr(0x6, 0x0, 0x8046e44, 0x83765b0), at 0xfec23e6f 
  [10] call_user_handler(0x6, 0x0, 0x8046e44), at 0xfec1a30e 
  [11] sigacthandler(0x6, 0x0, 0x8046e44, 0xf, 0x0, 0x807007), at 0xfec1a48e 
  ---- called from signal handler with signal 6 (SIGABRT) ------
  [12] __lwp_kill(0x1, 0x6), at 0xfec24aa7 
  [13] raise(0x6), at 0xfebd1217 
  [14] abort(0xfed9ff44, 0x80470e8, 0xfeffdcc8, 0x80470e8, 0x8617e20, 0x8d29750), at
0xfebb15c0 
  [15] __Crun::pure_error(0x8d29750, 0x8047164, 0x1, 0x0, 0x0, 0x0), at 0xfec6656a 
  [16] Sync::lock(this = 0x8047164, type = Exclusive), line 58 in "Sync.cpp"
  [17] MemMgr::alloc(this = 0x8d296c0, length = 48), line 328 in "MemMgr.cpp"
  [18] MemMgr::allocateDebug(this = 0x8d296c0, size = 24, fileName = 0x8b3ae34
"MemoryManager.h", line = 48), line 539 in "MemMgr.cpp"
  [19] MemMgrAllocateDebug(s = 24U, file = 0x8b3ae34 "MemoryManager.h", line = 48), line
124 in "MemMgr.cpp"
  [20] operator new(s = 24U), line 48 in "MemoryManager.h"
  [21] std::basic_filebuf<wchar_t,std::char_traits<wchar_t> >::close(0xfede02d0), at
0xfecec0c6 
  [22] std::basic_filebuf<wchar_t,std::char_traits<wchar_t>
>::~basic_filebuf(0xfede02d0), at 0xfece8917 
  [23] std::__Wide_Init::~__Wide_Init(0x8047284), at 0xfed63af2 
  [24] __SUNW_fini_iostreams(0xfef001c0, 0xfec52218, 0xfed9ff44, 0xfef00200, 0xfec52218,
0xfec4d000), at 0xfed6a87b 
  [25] __Cimpl::cplus_fini(0xfec4d000, 0xfebb1d7d, 0x80474c8, 0x804734c, 0xfec4d000,
0xfec52200), at 0xfec663eb 
  [26] __cplus_fini_at_exit(0xfeffb818, 0x8374dfc, 0x0, 0x0, 0x0, 0x0), at 0xfed7c9b2 
  [27] exit(0x0, 0x0), at 0xfeba4a32 
  [28] main(argc = 7, argv = 0x8047390), line 4638 in "mysqld.cc"

Note 1: this only happens when I have compiled with --with-debug.

Note 2: The call stack contains two crashes. The initial crash in Falcon code and a
second crash in the server code. The crash in the server code is reported in bug #37961
(http://bugs.mysql.com/bug.php?id=37961).

How to repeat:
1. Configure MySQL with Falcon for Sun Studio compiler with the --with-debug option.

2. Compile

3. Run mysql_install_db

To verify whether the MemMgr was a valid object or had already been destructed at the time
of the crash I added a fprintf to the constructor and destructor of MemMgr and an assert
on a status variable for whether or not the destructor had been run on it. The result:

Installing MySQL system tables...
MemMgr::Mgr(int,int,int)
MemMgr::Mgr(int,int,int)
MemMgr::~MemMgr
MemMgr::~MemMgr
Assertion failed: init == 6, file MemMgr.cpp, line 340

So the cause for this crash is basically that we are trying to allocate a new object from
an already destructed Memory Manager (which also was Vlad's theory).

Fix is in http://lists.mysql.com/commits/50993
Pushed to falcon-team

A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/51134

2770 Vladislav Vaintroub	2008-08-07 [merge]
      merge

A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/51135

2770 Vladislav Vaintroub	2008-08-07 [merge]
      merge

This fix is in version 6.0.7

Pushed into 6.0.7-alpha  (revid:cbell@mysql.com-20080822132131-uveo6wiuecy6m2b8) (version
source revid:cbell@mysql.com-20080822132131-uveo6wiuecy6m2b8) (pib:3)

This solution is temporary. The real problem is outlined in Bug#38598.  Once we can use
the Falcon memory manager without overriding global new and delete, then the Falcon
Memory Manager should be reverted back to only using itself.  It should be used within
Falcon only when it is instantiated.  This patch makes it switch automatically to use
plain malloc() & free() if the memory manager is not alive.  This solution has caused
follow-on problems when a Jstring was allocated before the MemMgr was alive, but freed
after.  See the latest patch for Bug#38041; http://lists.mysql.com/commits/52288.

Pushed into 6.0.7-alpha  (revid:vvaintroub@mysql.com-20080806115321-60zhg0yehaxhca7c)
(version source revid:john.embretsen@sun.com-20080724122511-9c0oudz1xrdrs6y6) (pib:3)

A note has been added to the 6.0.7 changelog: 

When using mysql_install_db on MySQL built with Sun Studio 12 with the --with-debug
option enabled, the server would crash