MySQL Bugs: #42475: Falcon can crash in Sync::findLocks after a timeout.

Bug #42475	Falcon can crash in Sync::findLocks after a timeout.
Submitted:	30 Jan 2009 12:51	Modified:	15 May 2009 13:02
Reporter:	Vladislav Vaintroub	Email Updates:
Status:	Closed	Impact on me:	None
Category:	MySQL Server: Falcon storage engine	Severity:	S3 (Non-critical)
Version:	6.0.9	OS:	Any
Assigned to:	Vladislav Vaintroub	CPU Architecture:	Any
Tags:	F_SYNCHRONIZATION

Description:
When running falcon_pagesize_32k RQG test on Windows, mysqld crashed with following
stacktrace

0077C9C9    mysqld.exe!Sync::findLocks()[sync.cpp:107]
0078887F    mysqld.exe!Thread::findLocks()[thread.cpp:410]
00776164    mysqld.exe!SyncObject::findLocks()[syncobject.cpp:723]
0078887F    mysqld.exe!Thread::findLocks()[thread.cpp:410]
007766E4    mysqld.exe!SyncObject::stalled()[syncobject.cpp:699]
00776C0A    mysqld.exe!SyncObject::wait()[syncobject.cpp:678]
00776E6E    mysqld.exe!SyncObject::lock()[syncobject.cpp:449]
0077C8BF    mysqld.exe!Sync::lock()[sync.cpp:59]
0078497D    mysqld.exe!Table::insertIndexes()[table.cpp:1268]
00784E47    mysqld.exe!Table::insert()[table.cpp:3058]
00777248    mysqld.exe!StorageDatabase::insert()[storagedatabase.cpp:267]
00770D32    mysqld.exe!StorageTable::insert()[storagetable.cpp:110]
00769CAF    mysqld.exe!StorageInterface::write_row()[ha_falcon.cpp:1159]
00449D5E    mysqld.exe!handler::ha_write_row()[handler.cc:5442]
0054989D    mysqld.exe!write_record()[sql_insert.cc:1610]
0054F149    mysqld.exe!mysql_insert()[sql_insert.cc:836]
004BE9E1    mysqld.exe!wait_if_global_read_lock()[lock.cc:1324]

How to repeat:
runall.pl \ 
--basedir=path/to/mysql-basedir \
 --mysqld=--loose-innodb-lock-wait-timeout=1 \ 
 --mysqld=--table-lock-wait-timeout=1 \ 
 --mysqld=--loose-falcon-lock-wait-timeout=1 \ 
 --mysqld=--loose-falcon-debug-mask=2 \ 
 --mysqld=--skip-safemalloc \ 
 --grammar=conf/falcon_pagesize.yy \ 
 --mysqld=--falcon-page-size=32K \ 
 --gendata=conf/falcon_pagesize32K.zz \ 
 --reporters=Deadlock,ErrorLog,Backtrace,Recovery,WinPackage \ 
 --duration=1200 \ 
 --mysqld=--log-output=file \ 
 --queries=100000 \ 
 --engine=falcon

A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/64626

2990 Vladislav Vaintroub	2009-01-30
      Bug#42475 crash in Sync::findLocks.
      
      Falcon would sporadically crash in high concurrency situations,when some threads are stalled waiting for a lock. The crash comes from lock analysis function Sync::stalled(), that prints list of locks and wait queue.
      
      Problem: 
      SyncObject::wait() does not reset thread->lockPending is not NULL in timeout situations.
      So it can happen that lockPending points to invalid memory for some Thread object. 
      When Sync::stalled() is trying to analyze the locks, it will access lockPending and crash.
      
      Fix:
      Set  thread->lockPending b NULL before SyncObject::wait() throws exception in timedout().

A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/64628

2990 Vladislav Vaintroub	2009-01-30
      Bug#42475 crash in Sync::findLocks.
      Falcon would sporadically crash in high concurrency situations,when some threads are stalled waiting for a lock. The crash comes from lock analysis function Sync::stalled(), that prints list of locks and wait queue.
      
      Problem: 
      SyncObject::wait() does not reset thread->lockPending to NULL in timeout situations. So it can happen that lockPending points to invalid memory for some Thread object.  When Sync::stalled() is trying to analyze the locks, it will access lockPending and crash.
      
      Fix:
      Set  thread->lockPending b NULL before SyncObject::wait() throws exception in timedout().

The change looks correct. OK to push

A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/64804

2990 Vladislav Vaintroub	2009-02-01
      Bug#42475 crash in Sync::findLocks.
      Falcon would sporadically crash in high concurrency situations,when some threads are stalled waiting for a lock. The crash comes from lock analysis function Sync::stalled(), that prints list of locks and wait queue.
      
      Problem: 
      SyncObject::wait() does not reset thread->lockPending to NULL in timeout situations. So it can happen that lockPending points to invalid memory for some Thread object.  When Sync::stalled() is trying to analyze the locks, it will access lockPending and crash.
      
      Fix:
      Set  thread->lockPending to NULL before SyncObject::wait() throws exception in timedout().

The second patch, setting thread->lockPending to NULL before SyncObject::wait() throws an exception in timedout(), is also approved.

Pushed into 6.0.10-alpha (revid:alik@sun.com-20090211182317-uagkyj01fk30p1f8) (version source revid:vvaintroub@mysql.com-20090201205427-yzedgslc2jqiu626) (merge vers: 6.0.10-alpha) (pib:6)

An entry has been added to the 6.0.10 changelog: 

When running Falcon during a very high concurrency workload, mysqld could fail.