Bug #43719 Falcon: performance regression in sysbench test OLTP_RW
Submitted: 18 Mar 2009 4:46 Modified: 26 May 2010 17:52
Reporter: Alexey Stroganov Email Updates:
Status: Unsupported Impact on me:
None 
Category:MySQL Server: Falcon storage engine Severity:S5 (Performance)
Version:6.0.10-release OS:Any
Assigned to: Kelly Long CPU Architecture:Any
Tags: F_PERFORMANCE

[18 Mar 2009 4:46] Alexey Stroganov
Description:
Performance of 6.0.10 binary in sysbench OLTP_RW test is notable worse
than in 6.0.9 especially with higher number of threads.

# Test: OLTP_RW
#
# Server 1 - 6.0.9-alpha
# Server 2 - 6.0.10-alpha
#
#            Server 1 Server 2
# Thread       FALCON   FALCON
          4   1104.83   939.59
         16    841.68   137.01
         64    404.43    45.38
        128    418.83    30.62
        256    412.99    22.25

From sar output it looks that this slowdown was caused by some locking
that was added after 6.0.9 because there is almost no activity at the user side.

                CPU     %user     %nice   %system   %iowait    %steal     %idle

Run with 256 threads for 6.0.9/Falcon:
Average:        all     48.67      0.00      8.02      0.42      0.00     42.88

Run with 256 threads for 6.0.10/Falcon:
Average:        all      2.62      0.00      0.83      0.04      0.00     96.51

Also below is output of oprofile for runs with 256 threads for 6.0.9 and 6.0.10.
Set of functions and their percentage are pretty similar (except SyncObject::grantLocks) but number of samples is differ by a factor of ~20.

6.0.9
-----
samples  %        symbol name
104987   40.2752  SyncObject::lock(Sync*, LockType, int)
34061    13.0665  SyncObject::unlock(Sync*, LockType)
13599     5.2169  DeferredIndex::scanIndex(IndexKey*, IndexKey*, int, Bitmap*)
11770     4.5152  Index::scanIndex(IndexKey*, IndexKey*, int, Transaction*, Bitmap*)
7323      2.8093  Transaction::visible(Transaction*, unsigned int, int)
5421      2.0796  DeferredIndex::deleteNode(IndexKey*, int)
4914      1.8851  Sync::lock(LockType)
4601      1.7650  my_hash_sort_simple
2657      1.0193  Sync::Sync(SynchronizationObject*, char const*)
2173      0.8336  Sync::~Sync()
1992      0.7642  ptr_compare_0
1955      0.7500  Record::addRef()
1576      0.6046  Bitmap::nextSet(int)
1513      0.5804  Table::fetch(int)
1509      0.5789  Transaction::initialize(Connection*, unsigned int)
1471      0.5643  SerialLogRecord::getInt()
1375      0.5275  Transaction::releaseDependency()
1253      0.4807  SyncObject::grantLocks()

6.0.10
------
4975     37.7037  SyncObject::lock(Sync*, LockType, int)
1602     12.1410  SyncObject::unlock(Sync*, LockType)
590       4.4714  DeferredIndex::scanIndex(IndexKey*, IndexKey*, int, Bitmap*)
496       3.7590  Index::scanIndex(IndexKey*, IndexKey*, int, Transaction*, Bitmap*)
291       2.2054  Transaction::visible(Transaction*, unsigned int, int)
273       2.0690  SyncObject::grantLocks()
240       1.8189  Sync::lock(LockType)
200       1.5157  RecordScavenge::inventoryRecord(Record*)
161       1.2202  DeferredIndex::deleteNode(IndexKey*, int)
146       1.1065  Cache::fetchPage(Dbb*, int, PageType, LockType)
133       1.0080  MemMgr::blockSize(void*)
130       0.9852  Sync::~Sync()
99        0.7503  Sync::Sync(SynchronizationObject*, char const*)
85        0.6442  Bitmap::nextSet(int)
76        0.5760  ptr_compare_0

How to repeat:
 - Download and build the latest sysbench from sysbench.sf.net
 - prepare database 

sysbench \
--test=oltp \
--oltp-table-size=1000000 \
--oltp-dist-type=uniform \
--init-rng=on \
--mysql-user=root  \
--mysql-table-engine=falcon \
--mysql-db=test prepare

  - run the test 
sysbench  \
--oltp-table-size=1000000  \
--oltp-dist-type=uniform \
--init-rng=on \
--mysql-user=root \
--mysql-socket=/tmp/mysql.sock \
--mysql-db=test  \
--test=oltp \
--max-requests=0 \
--max-time=300 \
--oltp-read-only=off \
--num-threads=<4,16,64,128,256> \
run