Bug #15815 Very poor performance with multiple queries running concurrently
Submitted: 16 Dec 2005 23:18 Modified: 19 Jun 2010 0:01
Reporter: Peter Zaitsev (Basic Quality Contributor) Email Updates:
Status: Closed Impact on me:
None 
Category:MySQL Server: InnoDB storage engine Severity:S2 (Serious)
Version:4.0.27,4.1, 5.0, 5.1 OS:Linux (Linux x86_64)
Assigned to: Sunny Bains CPU Architecture:Any

[16 Dec 2005 23:18] Peter Zaitsev
Description:
It is not clear at this point how wide scope of the systems is affected. It is likely to affect all versions. 

It is happenin on RH AS 4  Linux with 4 Intel Xeon 64bit CPUs with  HT   (8 logical)

The problem is:  Query starts to take significantly longer if multiple queries are executed at once:

select sql_calc_found_rows  * from b limit 5;

1   client    -    4 sec 
4   clients  -  70 sec 

Note as there are 4 CPUs it  would be expected 4 clients to complete close to 5 sec or so, running in parallel.    The situation gets even worse with more clients. 

How to repeat:
CREATE TABLE `b` (
  `child_id` int(10) unsigned NOT NULL default '0',
  `b` char(20) default NULL,
  KEY `child_id` (`child_id`)
) ENGINE=InnoDB

Populate   table with 100.000+ of rows, but make sure it still fits in buffer pool.

insert into b (child_id) values (0);
insert into b (child_id) values (0);
insert into b (child_id) values (0);
insert into b (child_id) values (0);
insert into b (child_id) values (0);

insert into b select * from b;
....
insert into b select * from b;    (repeat while you get enough rows)

Create 1.sql file:

use test;
select sql_calc_found_rows  * from b limit 5;
select sql_calc_found_rows  * from b limit 5;
select sql_calc_found_rows  * from b limit 5;
select sql_calc_found_rows  * from b limit 5;
select sql_calc_found_rows  * from b limit 5;

Run 1 copy:

create 1.sh:

cat 1.sql | time mysql  >  /dev/null

Run 4 copies: 

create 4.sh:

./1.sh &
./1.sh &
./1.sh &
./1.sh &

Run 1.sh and 4.sh to see what happens for 1 and 4 connections.
[18 Dec 2005 0:48] Heikki Tuuri
Peter,

please post a few representative:

SHOW INNODB STATUS\G

Regards,

Heikki
[21 Dec 2005 22:10] Heikki Tuuri
Osku's upcoming sync0arr.c patch may fix this.
[6 Jan 2006 1:25] Brent Nelson
In my tests (using Peter's test case):

With 4 visable CPUs on a 4 physical CPU server
1 thread took 0:10 minutes
4 threads took 4:00 minutes
8 threads took 7:56 minutes
16 threads took 15:50 minutes

With 8 visable CPUs on a 4 physical CPU server (the same server - with hyperthreading)
1 thread took 0:10 minutes
4 threads took 6:30 minutes
8 threads took 20:12 minutes
16 threads took 40:43 minutes

So, more CPUs does seem to make the situation worse, as Peter predicted.
[6 Jan 2006 5:38] Brent Nelson
I tried running the same test, booting the server without an SMP kernel (one visable CPU).

I got the following results:

1 thread took 0:11 minutes
2 threads took 0:22 minutes
4 threads took between 0:25 and 0:44 minutes 
(some threads were faster than others)
8 threads took between 0:28 and 1:30 minutes
16 threads took between	1:38 and 2:36 minutes 
( 1 thread exited with lost connection to mysql during query )

So, a single CPU performed much faster than 4 physical CPUs, or 8 virtual CPUs.  Query times on a single CPU are close to linear.
[6 Jan 2006 5:39] Brent Nelson
I have access to 8 physical processor EM64T servers that I can run these tests on  as well, if that helps us test with large SMP systems.
[13 Jan 2006 1:07] Heikki Tuuri
Osku's patch seems to be working now.
[16 Jan 2006 16:44] Heikki Tuuri
Vadim found yet another assertion failure in the patch. Putting back to the 'In progress' status.
[19 Jan 2006 0:40] Heikki Tuuri
Heikki has contributed a fixed sync0arr.c.
[22 Jan 2006 10:36] Vadim Tkachenko
Heikki,

I did profiling of mutex waiting:

my profiling shows that all waits coming from (time in ms)
84,383   [7] row_search_for_mysql [row0sel.c]

deeply:
41,645 | [6] sel_restore_position_for_mysql [row0sel.c]
42,266 | [33] mtr_commit [mtr0mtr.c]

mtr_commit calls:
42,268 | [32] mtr_memo_slot_release [mtr0mtr.c] 
 -> 42,268 | [31] buf_page_release [sync0sync.ic]

sel_restore_position_for_mysql calls
 41,645 | [5] btr_pcur_restore_position [btr0pcur.c]
  ->  41,645 | [4] buf_page_optimistic_get_func [sync0sync.ic]

so, as I guessed before, there is a problem with intensive
buffer_pool locking / unlocking.
[31 Jan 2006 16:39] Heikki Tuuri
Vadim,

thank you. Looks like a very bad case of a cache line ping-pong or a memory bus locking problem.

Regards,

Heikki
[17 Feb 2006 13:30] Vadim Tkachenko
Heikki,

this bug in state Patch appproved.
Which patch do you mean?

If patch with sync_array - this patch does not help.
If another, could you send us patch ?

Thanks,
Vadim.
[17 Feb 2006 22:37] Heikki Tuuri
Vadim,

it is Osku's patch that is approved. The buf pool sync mutex still has to be solved. I will not put the bug to the 'Documenting' status until we know everything.

I am sorry for the delay, I have been in a flu this week.

Regards,

Heikki
[11 Mar 2006 2:50] Scott Marlowe
Hey guys, I was browsing the bugs database, and wanted to comment on this issue as regards hyperthreading.

Hyperthreading is generally a losing proposition on database machines.  Oracle and PostgreSQL suffer from much worse performance when hyperthreading is enabled as well.

Quite simply, hyperthreading tries to make one processor act like two, but those two pipelines share about 80% of the same pipelines AND the same caches.  

So, I would recommend NOT testing with hyperthreading turned on, at least for the initial comparisons and what not.

Also, if someone has a multicore opteron, those are quite worth testing.  each core gets its own cache, and the memory bandwidth is quite good.  Also, a multiple CPU Sun box would be a better choice.  Intel SMP machines tend to be cache / memory speed bound, and hyperthreading just makes this worse.
[11 Mar 2006 8:52] Heikki Tuuri
I ran the sysbench 0.4.4-sp insert test yesterday on Linux/4 X AMD64, and the first serious drop of performance happened at 64 threads. It is the semaphore waits and sync0arr.c that clearly are the problem.

Since the upcoming 5.1.8 contains Osku's fixes, that version will perform better in sysbench.
[31 Mar 2006 16:01] Heikki Tuuri
'Patch pending' describes the state of this better.

Osku's patch removes much of the contention in 5.1, but I need to do more work. Unfortunately, for the past 4 weeks, I have been working on other bug reports.
[5 Apr 2006 17:22] Dmitriy Kropivnitskiy
OK, here is another benchmark for this case. I am running community MySQL 5.0.18 on RH ES 4 update 3). This is a dual CPU system with 2 dual core Opteron CPUs running 2.4 Ghz (virtual 4 CPUs). It has 16GB RAM. I have changed the tests slightly, since original scripts only gave me times for each mysql process and not for the whole cycle. So I used following script x.sh

for ((i=0; i<$1; i++)); do
        cat 1.sql | mysql test >/dev/null &
done
wait

And ran is as 
time sh x.sh <number of threads>

Also, I turned off the query cache (just in case) and user following innodb settings
innodb_commit_concurrency=1
innodb_thread_concurrency=2

So here are the results:

1 thread
real    0m0.009s
user    0m0.002s
sys     0m0.007s

2 threads
real    0m0.013s
user    0m0.009s
sys     0m0.008s

4 threads
real    0m0.015s
user    0m0.017s
sys     0m0.013s

8 threads
real    0m0.023s
user    0m0.030s
sys     0m0.026s

16 threads
real    0m0.035s
user    0m0.060s
sys     0m0.048s

32 threads
real    0m0.063s
user    0m0.128s
sys     0m0.094s

64 threads
real    0m0.129s
user    0m0.264s
sys     0m0.176s

128 threads
real    0m0.249s
user    0m0.503s
sys     0m0.391s

256 threads
real    0m0.493s
user    0m1.017s
sys     0m0.757s

512 threads
real    0m0.971s
user    0m2.061s
sys     0m1.544s

1024 threads
real    0m1.982s
user    0m4.177s
sys     0m3.157s

2048 threads
real    0m3.789s
user    0m7.843s
sys     0m6.075s

4096 threads
real    0m6.986s
user    0m14.407s
sys     0m11.427s

8192 threads
real    0m13.685s
user    0m27.997s
sys     0m22.275s

Seems like the execution time is going up in a very linear fashion, as in executing twice as many threads takes twice as long no matter what is the number of threads.
[5 Apr 2006 18:03] Vadim Tkachenko
Heikki,

I tested 5.0.19 and 5.1.9 with Peter's workload:

5.0.19 1 thread 09.18s, 4 thread 1:08.79
5.1.9  1 thread 10.11s, 4 thread 1:21.79

So in 5.1.9 things go even worse.
[5 Apr 2006 18:24] Vadim Tkachenko
Heikki,

Should we still use innodb_thread_concurrency=1 in 5.1.9 ?
[5 Apr 2006 19:24] Peter Zaitsev
Dmitriy,

You some decent test run time. In your case  you seems to be using very small table or something, so it takes only 1ms for single thread.  I guess most of this time is consuming my starting mysql client and connecting.   I would recommend to have it at least 5 seconds for conclusive results.

Not innodb_thread_concurrency=1 is known workaround. However it  very much limits resources avaible to Innodb - if you're having heavy Innodb workload this will mean you will barely use more than 1 CPU on your system.
[7 Apr 2006 13:11] Vadim Tkachenko
Heikki, 

here is additional info, 5.1.9

table 

CREATE TABLE IF NOT EXISTS `$tableName` (
    `id` int(10) unsigned NOT NULL auto_increment,
    `name` varchar(64) NOT NULL default '',
    `email` varchar(64) NOT NULL default '',
    `password` varchar(64) NOT NULL default '',
    `dob` date default NULL,
    `address` varchar(128) NOT NULL default '',
    `city` varchar(64) NOT NULL default '',
    `state_id` tinyint(3) unsigned NOT NULL default '0',
    `zip` varchar(8) NOT NULL default '',
    `country_id` smallint(5) unsigned NOT NULL default '0',
    PRIMARY KEY  (`id`),
    UNIQUE KEY `email` (`email`),
    KEY `country_id` (`country_id`,`state_id`,`city`)
    )
    
queries:    
UPDATE $tableName SET name=%s WHERE id between %d and %d

Threads	queries/sec
1	130,467
4	172,323
16	40,962
64	2,823
256	  0,232
1024		Timeout

With innodb_thread_concurrency=1 I have stable result ~ 140 q/s

In additional in 5.1.9 I observe the table is full errors:

Query UPDATE normal SET name='jul55453em643z32r52efj4kr7vsl0cc' WHERE id between 860642 and 860742 error: The table 'normal' is full (1114)
Query UPDATE normal SET name='2n36g2iff708hysckrpzmn' WHERE id between 903986 and 904086 error: The table 'normal' is full (1114)
Query UPDATE normal SET name='2nag01jkm65ad9xq6jovw09vz3wnwa663tnydyhw3o3rht' WHERE id between 485699 and 485799 error: The table 'normal' is full (1114)
Query UPDATE normal SET name='ezwxx4sbggudud0f7b36mrairi1coyzcfgvy' WHERE id between 728647 and 728747 error: The table 'normal' is full (1114)
Query UPDATE normal SET name='3q3spxxyn53jcfafikwtexnq2n1o' WHERE id between 279625 and 279725 error: The table 'normal' is full (1114)

mysql> show innodb status\G
*************************** 1. row ***************************
  Type: InnoDB
  Name: 
Status: 
=====================================
060407 14:59:52 INNODB MONITOR OUTPUT
=====================================
Per second averages calculated from the last 46 seconds
----------
SEMAPHORES
----------
OS WAIT ARRAY INFO: reservation count 10076104, signal count 282148
--Thread 2142886832 has waited at btr0cur.c line 366 for 0.00 seconds the semaphore:
Wait array cell state 1
S-lock on RW-latch at 0xb6fc1cbc created in file dict0dict.c line 3648
a writer (thread id 2258475952) has reserved it in mode  wait exclusive
number of readers 1, waiters flag 1
Last time read locked in file btr0cur.c line 366
Last time write locked in file btr0cur.c line 359
--Thread 2304347056 has waited at btr0cur.c line 366 for 0.00 seconds the semaphore:
Wait array cell state 1
S-lock on RW-latch at 0xb6fc1cbc created in file dict0dict.c line 3648
a writer (thread id 2258475952) has reserved it in mode  wait exclusive
number of readers 1, waiters flag 1
Last time read locked in file btr0cur.c line 366
Last time write locked in file btr0cur.c line 359
--Thread 2270190512 has waited at btr0cur.c line 366 for 0.00 seconds the semaphore:
Wait array cell state 1
S-lock on RW-latch at 0xb6fc1cbc created in file dict0dict.c line 3648
a writer (thread id 2258475952) has reserved it in mode  wait exclusive
number of readers 1, waiters flag 1
Last time read locked in file btr0cur.c line 366
Last time write locked in file btr0cur.c line 359

... 163 times ...

--Thread 2270190512 has waited at btr0cur.c line 366 for 0.00 seconds the semaphore:
Wait array cell state 1
S-lock on RW-latch at 0xb6fc1cbc created in file dict0dict.c line 3648
a writer (thread id 2258475952) has reserved it in mode  wait exclusive
number of readers 1, waiters flag 1
Last time read locked in file btr0cur.c line 366
Last time write locked in file btr0cur.c line 359
... ..

--Thread 2156608432 has waited at btr0cur.c line 366 for 0.00 seconds the semaphore:
Wait array cell state 1
S-lock on RW-latch at 0xb6fc1cbc created in file dict0dict.c line 3648
a writer (thread id 2258475952) has reserved it in mode  wait exclusive
number 
1 row in set, 1 warning (0.13 sec)
[7 Apr 2006 14:45] MySQL Verification Team
Vadim,

What you experience as "table is full" is actually behaviour described  in the bug #18828.
[15 May 2006 15:34] Davide Ferrari
I'm sorry if I may be offtopic in this BR but I'm doing some performance testing on various SMP machines (Xeon adn Opteron) with sysbench OLTP benchmark...and I'm a bit disappointed by the results. I'm using MySQL 5.0.21 on a 2.6.16 Gentoo system, I've tried both Gentoo compiled and precompiled Mysql distribution, and the Xeon machine (dual xeon 3.8Ghz with 8GB RAM) acts too much better then the dual Opteron (Opteron 275 with 16GB RAM)...better as 640 tps (Xeon) versus 110 (!!!) tps for the Opteron. Is this normal or has something to do with this bug? I can't really find any answer on the web til now :(
[15 May 2006 16:30] Vadim Tkachenko
Davide,

I think something is wrong with setups.
We have a lot of results on opteron/xeon and usually mysql on opterons performs such good as on xeons.

I propose discuss it in emails.
[8 Jun 2006 15:52] Christian Hammers
This is still "patch pending".. any progress?

bye,

-christian
[8 Jun 2006 16:03] Heikki Tuuri
Christian,

Osku's patch in 5.1 relieves this problem.

For 5.0, a partial workaround is to set

innodb_thread_concurrency=4

or even lower.

I am painfully aware of this bug report, and I am working on this.

Regards,

Heikki
[31 Jul 2006 10:55] Daniel Saito
Aside of the work around in 5.0, are there any plans of working on a patch for 5.0?
[9 Aug 2006 14:26] Heikki Tuuri
Daniel,

we have not decided if we will put some more drastic fix to 5.0. Since 5.0 is the production version, we should avoid making big changes like this.

I intend to assign our newly hired developer to work on this problem soon.

Regards,

Heikki
[10 Aug 2006 16:44] Heikki Tuuri
My current thought on fixing this is that we introduce one 'event' per InnoDB semaphore and we let the 'sync array' to be just for monitoring hung threads. This is an idea from Vadim Tkachenko. The problem in the old sync array seemed to be that if every waiting thread has its own semaphore, and we release all threads at the same time, operating systems do not schedule the released threads efficiently. It is more natural that there is just one event per InnoDB mutex or rw-lock. When the mutex or rw-lock is released, we set that event, and waiting threads can proceed.

The reason for not creating one event per InnoDB semaphore was that Windows 1995 could not create that many events. Since no one runs big databases on Win 95 any more, we can ignore that problem now.
[10 Aug 2006 20:02] Vadim Tkachenko
Heikki,

I think improving of sync_array will not help there!!

The problem is CPU time is spent in SPIN_LOOP.
Count of os_event is very small, so sync_array is called very rarely and is absolutely uncritical here.
[15 Aug 2006 11:16] Heikki Tuuri
Please note that I have bundled two different bugs to this bug report:

1) 'Thread thrashing' with > 50 concurrent connections under an update-intensive workload. 
       Suggested fix: introduce one event per InnoDB semaphore;

2) Buffer pool mutex contention under >= 4 big concurrent CPU-bound SELECT queries.
       Suggested fix: replace the mutex by one mutex protecting the 'flush list' (and the free list) and several mutexes protecting portions of the buffer pool, where we keep several indivudual LRU lists of pages.
[15 Aug 2006 14:43] Heikki Tuuri
Assigning this to Sunny. This is a larger project than just a normal bug fix.
[31 Aug 2006 18:00] James Day
An experimental partial patch for 5.0.22 is available and being discussed at http://www.mysqlperformanceblog.com/2006/07/28/returning-to-innodb-scalability/ . The official patch, when it is ready, is likely to more thoroughly address a range of mutex-related issues. You should not use this patch on mission-critical servers, but I'm mentioning it for anyone who can't use the established workaround and needs to have something that may be more effective. Do not use it at all if SHOW MUTEX STATUS does not show exactly the same mutex issue for buf_pool->mutex). The patch only addresses that specific problem. Once an official patch is available you should migrate to that as soon as practical.

This unofficial patch is not supported in any way by MySQL.
[7 Sep 2006 19:44] James Day
There is also http://www.mysqlperformanceblog.com/2006/09/07/internals-of-innodb-mutexes/ .

Again, this is an unofficial patch in no way supported by MySQL, but may be of use until a more comprehensive solution is avaialble.
[11 Sep 2006 17:29] Andrew Houghton
Can we get a status report from MySQL on this?  It's been nearly a month since the last note from Heikki.  

In particular, for planning purposes, I would *love* to know if there are any plans to backport a fix from 5.1.  I'm very close to putting new multi-core multi-proc machines into production, and it seems fairly clear that I'm going to run into this issue immediately.
[11 Sep 2006 18:28] James Day
Andrew, we're investigating what we can do to get this sorted out more quickly. We may support the patches I referenced earlier. Still in very active discussion so I can't say more about what will actually be done at this point. 

Up to 4 cores are often reported as not affected, but some with 4 have reported trouble. One customer had trouble with 8 cores but not with 4. If your CPUs have hyperthreading, turn that off. Experiment with innodb_thread_concurrency to find the best value, which varies. 4 seems always safe. Disk-limited load can go higher than CPU-limited load. How much higher can only be determined experimentally.

Sorry, I can't say what versions will get any fix we provide. Still undecided. We do know that versions 4.1, 5.0 and 5.1 are affected. It seems likely that 4.0 and 3.23 are affected, but very unlikely that changes will be made for these two. So far no customers have reported problems on 4.0 or 3.23.

Besides that, all I can really say today is watch this space for developments over the next two weeks or so, as the discussions and actions that follow them start to happen. This is not a guaranteed timeline, just an estimate. Don't be surprised it it takes longer to get a final result.

James Day, Support Engineer, MySQL AB
[13 Sep 2006 20:55] Andrew Houghton
Good enough; your suggestions give me hope that we'll be able to use most of the processing power of the new boxes during an interim period (they're dual Opteron 275 boxes, and we'll be running 5.0.x), and at the very least we have some ideas of what to tweak to make them stable.

Thanks!
[14 Sep 2006 3:52] Don MacAskill
Can someone update us about 5.1.x?  Heikki mentions Osku's patch a few times, but it's not obvious whether a version of 5.1 has been released with it, and if not, where we can find said patch.

And finally, does it actually solve the problem in 5.1?  Or just make it less severe?  Is 5.1 "fixed" ?

Thanks.
[14 Sep 2006 5:51] James Day
Don, no official patch has been released and what will be developed and released when hasn't been decided. Watch this space is all I can usefully say right now. The patch I linked to above was reported by one user as producing an 80% improvement, so that sounds largely fixed. I doubt that an official patch would do less well.
[3 Oct 2006 1:14] Sunny Bains
This bug report has been split into two. The 'thread thrashing' [ see Heikki's
comment 15 Aug 13:16 ] problem is now filed under Bug# 22868.
[3 Oct 2006 3:45] Yasufumi Kinoshita
I have made the experimental patch (for 5.0.22..).
This patch makes the all of the "buf_pool->mutex" devided and optimized.
And it is significantly effective for scaling performance along with the number of CPUs.
(I have confirmed it up to at least 8CPUs.)
Will this method solve this bug ?

...And, How do I attach the patch to this report ?
[3 Oct 2006 7:45] Valeriy Kravchuk
Yasufumi,

If it is <200K in size, you can use the Files tab to upload your patch.
[3 Oct 2006 8:39] Yasufumi Kinoshita
Valeriy,

Of course, I've tried it.
But the system says ..
"Only registered developers and the original submitter are allowed to attach files to the bug."
[4 Oct 2006 5:48] Timothy Smith
Yasufumi Kinoshita-san,

You may upload the patch (with anonymous FTP) to:

ftp://ftp.mysql.com/pub/mysql/upload/bug15815-kinoshita.diff

When it is uploaded, please let us know the name of the file and the file size.  I apologize for the inconvenience.

Regards,

Timothy
[5 Oct 2006 1:40] Yasufumi Kinoshita
Thank you, Tim.
No problem.

I have just uploaded "mysql-5.0.22_innodb-CPU-scale.patch.diff.gz" (9.8KB).

And..

< Outline of this patch (5.0.22) >

--All of "buf_pool->mutex" is devided to ..

  buf_pool->LRU_mutex                   (LRU list)
  buf_pool->flush_list_mutex            (flush list)
  buf_pool->free_mutex                  (free list)
  buf_pool->page_hash_cell_mutex_arr[n] (page hash (for each cells))
  block->blockmutex                     (each control blocks)
  buf_pool->mutex                       (other parameters: mainly for aio consistency)

  And only the necesaries are used in each cases.

--To avoid the deadlock, mutex_enter_xx() is executed certainly in this order.

--The consistency of "LRU list" and "page hash" is protected by "buf_pool->LRU_mutex"

--These are not implemented yet...
    1. buf_pool->awe_LRU_free_mapped_mutex (Correspondence to AWE)
            But, I don't build a binary for Windows..

    2. Appropriate change in "#ifdef UNIV_DEBUG ... #endif"
            It doesn't work in debug mode. 

    3. Kind comments ... :-)

    4. Division and load-balancing of the LRU list.
            The parallelism of buf_pool->LRU_mutex might be a little insufficient still. 

    5(not necessary?). Improvement of kernel_mutex'es parallelism
            It might be necessary too, if the system have many CPUs (16? 32?).
[5 Oct 2006 17:38] Timothy Smith
Yasufumi-san,

Thank you for your contribution.  While there is still some debate about how to best fix this, I do hope that your patch will be found useful as a significant amount of work and preparation has been put into it.  It surely provides a real benefit for multi-cpu concurrency performance.

I have made it available for general download at:

ftp://ftp.mysql.com/pub/mysql/mysql-5.0.22_innodb-CPU-scale.patch.diff.gz

Kind regards,

Timothy
[12 Oct 2006 18:20] luojia chen
Hi,
I'm trying to download the patch from:
ftp://ftp.mysql.com/pub/mysql/mysql-5.0.22_innodb-CPU-scale.patch.diff.gz
But the page is not public available yet?
[13 Oct 2006 7:05] Timothy Smith
Hi.  I'm sorry, I typed the URL incorrectly.  It should be:

ftp://ftp.mysql.com/pub/mysql/download/mysql-5.0.22_innodb-CPU-scale.patch.diff.gz
[20 Oct 2006 18:56] James Day
Status update: the initial Oracle/InnoDB code has taken a couple of person-months to write but it's currently producing good results and is in test/QA/refinement work at Oracle/InnoDB. That suggests that there's a fair chance that it'll be production-ready by the end of the year. Possibly sooner, possibly later, since it depends on how the testing goes. It involves several critical core components of InnoDB so it's going to have to go slowly with lots of testing, lest it break things.

Please do remember that timelines are always estimates and the real time is "when it's ready". I can estimate based on experience but that's all it is.

James Day
Support Engineer, MySQL AB.
[21 Oct 2006 22:13] [ name withheld ]
Hello,

Is this issue specific to linux, or to hardware with multiple cpus?

We'll have this issue on a sun T1000 + solaris?
And what about a sun T1000 + linux?

thanx for your answers.
[23 Oct 2006 23:50] James Day
Status update: passed Oracle/InnoDB tests, waiting for final Oracle/InnoDB approval from Heikki before passing on to MySQL.

Sebastien, it potentially applies to all software and hardware when there is a high CPU-limited load. Whether it is an issue depends on the specific workload. I've seen published benchmarks showing MySQL having more scaling issues on Sun/Solaris Tnnnn hardware with multiple cores than on Opterons doing the same work with fewer cores. If you have not already seen it you may also find this article useful: http://developers.sun.com/solaris/articles/mysql_perf_tune.html .

If you are planning to use MySQL 5.1 I expect that there will be a solution by the end of the year. If you are planning to use MySQL 5.0 I suggest watching until the final decisions on including the fix in 5.0 and 4.1 have been made. This does cause (deliberate) crashes due to mutex timouts so we probably do have to fix it in 5.0 - 5.1 seems too far off for those with critical production systems, who proably can't prudently switch for a year.

It will be very interesting to see how MySQL scales with these changes in place.

James Day, Support Engineer, MySQL AB
[25 Oct 2006 15:26] Carl Dunham
I'd just like to throw in a vote for getting this prioritized for 5.0. 5.1 isn't even in RC yet, and this is an issue that hits a lot of high-end production systems running certified code *right now*.

Thanks!
[25 Oct 2006 17:49] Brent Nelson
It is good news that a fix has been implemented by the InnoDB team.  However, until it is available in the stable enterprise ready release of MySQL, it is not any good to us.

Anyone who is having a problem with this bug, I urge you to contact your account representative at MySQL and plead for this to be fixed in MySQL 5.0 as soon as possible.  It is important that we (the larger customers of MySQL) let the MySQL organization know how important this issue is to us.
[27 Oct 2006 7:57] James Day
Status update: It's currently planned to include fixes for bug #15815, bug #15868 and bug #22868 in 5.0 as well as 5.1.

James Day, Support Engineer, MySQL AB
[27 Oct 2006 8:12] James Day
Early builds: I'll announce it here when we have daily development builds including these fixes available. Note that these are not fully tested releases, just the developer's latest snapshot builds. Good enough for testing to discover any problems and confirm the fix, not recommended for full-scale production use by anyone not badly affected already. I'm not expecting this for a week or three.
[2 Nov 2006 18:24] Heikki Tuuri
I have now polished Sunny's patch.
[8 Nov 2006 14:45] Marko Mäkelä
The fix for 5.0 and 5.1 has been sent to MySQL AB.
[9 Nov 2006 19:33] Jeremy Cole
Kinda hard to use the MySQL intranet if you're not an employee isn't it?
[9 Nov 2006 19:35] James Day
Thanks, so it is. I'll do something a little different and make a more useful announcement...
[9 Nov 2006 19:38] MySQL Verification Team
Jeremy,

If I read it correctly, you would like to come back ..... ;o)
[9 Nov 2006 20:56] James Day
Source code for the first 5.0 build containing the fix for this bug is expected to be available from http://downloads.mysql.com/snapshots.php on Friday afternoon, US time. Please do not use this on critical production servers. Please do test it on non-critical servers.

5.1 is likely to follow within a few days. 4.1 is likely to follow only after release of 5.0 and 5.1 builds and some production experience.

At present the preliminary plan is not to release a 4.0 version. If you must
have this for the no longer fully supported 4.0 version and cannot upgrade,
please contact your sales contact or open a support issue requesting it and
saying why you need it and can't upgrade.

James Day
Support Engineer, MySQL AB
[11 Nov 2006 3:40] Paul DuBois
Noted in 5.0.30 (not 5.0.29), 5.1.13 changelogs.

InnoDB showed substandard performance with multiple queries running
concurrently.
[11 Nov 2006 3:43] Paul DuBois
Setting report to NDI pending decision about whether to fix in 4.1
[30 Nov 2006 1:20] James Day
MySQL Enterprise 5.0.30 incorporates the official patch for this and has been released. It can be obtained from http://enterprise.mysql.com . Please contact your sales person if you don't have access.

Those who are not Network/Enterprise or other Support customers can obtain the Enterprise source code from ftp://ftp.mysql.com/pub/mysql/src when it is released there.

James Day
Support Engineer, MySQL AB
[30 Nov 2006 18:58] Dathan Pattishall
I would love for this patch to go into 4.1 ASAP. It would really help me in some things that I'm doing. I'm working on internal channels to help facilitate this process.
[30 Nov 2006 18:59] Dathan Pattishall
I would love for this patch to go into 4.1 ASAP. It would really help me in some things that I'm doing. I'm working on internal channels to help facilitate this process.
[30 Nov 2006 19:04] MySQL Verification Team
I don't think that this fix should go into 4.1.

We have now observed few little problems on some installations, so it is possible that patch will still have to be perfected further.
[30 Nov 2006 19:20] David Pinard
Sinisa,
Is there a list of the "little problems" to help me determine if I should move to this build?  I am currently running an a custom build that includes a non-Mysql supported patch.

Thanks -Dave
[30 Nov 2006 19:31] MySQL Verification Team
No, there is no list yet.

We have yet to diagnose the first problem reported and in such a manner that customer's data is not revealed.

It will take some time, definitely, but the more testing is done by all of us, the better !!!!
[2 Dec 2006 8:01] James Day
Enterprise/Network and other Support customers who want this for 4.1 or 4.0 should approach their MySQL Sales contact to request a patched version, or open a support issue as an alternative route to the sales contact.

James Day, Support Engineer, MySQL AB
[15 Dec 2006 22:28] Nick Gaugler
So what's going the rate these days to purchase a patch?  $1?  $5?  $50?    Can we buy little cards at Walmart like we do for calling cards?  If we buy 5 do we get one free?  Will there be holiday specials as well?  I'd like to wait and buy a few patches when I can get a good deal on it.
[18 Dec 2006 18:14] James Day
Nick, the rates vary:

1. talk with your MySQL Sales contact if you want a build from MySQL once one is available (is for Enterprise now)
2. build from source
3. wait for the next Community binary release of the MySQL Server
4. wait for it to be incorporated in a build from your *nix distribution

Options 2 and 3 cost nothing; 4 may or may not, depending on your preferred distribution.
[22 Dec 2006 17:13] James Day
Status update: The fix appears to help but some users find that the problem moves elsewhere within the server, not always within InnoDB. Work to follow up those cases continues and it seems likely that further improvements, most likely for 5.1 and later only, may arrive. We'll decide based on what we find, what difference it seems to make and how risky it seems to be to incorporate it into what are supposed to be stable production releases like 5.0, and the very stable and low risk 4.1 and 4.0.

James Day, Support Engineer, MySQL AB
[10 Jan 2007 18:27] James Day
The source only MySQL Community Server 5.0.33 has been released today and contains this improvement. Binaries from MySQL will not be available for this Community server version. Source is available from http://dev.mysql.com/downloads/mysql/5.0.html .
[24 Jan 2007 17:46] Paul DuBois
A patch was backported to 4.0.29 and 4.1.23 as well.
I've added changelog entries for those versions.
[24 Jan 2007 21:32] Dathan Pattishall
Is this in 4.1.23 community edition? Is mySQL going to support a binary on the download site? Or is this just for Enterprise edition users?
[25 Jan 2007 4:49] James Day
Dathan,

I don't yet know whether there will be a Community binary release of this by MySQL. I'll post an update here when I do.

It should be available in BK now. We're just about to start getting things rolling on providing it for those with Enterprise/Network/Support subscriptions who have specifically requested it as a hotfix. The next other planned 4.1 release, for anyone, is a few months away.
[25 Jan 2007 19:50] James Day
Dathan, I passed this up the chain and here's a more official response to your question:

----

According to the MySQL Lifecycle Policy, no new Community releases of MySQL 4.0 or 4.1 are offered, as we are past the regular lifecycle of those releases. For those Community users who wish to get a fix to the InnoDB scalability bug http://bugs.mysql.com/bug.php?id=15815 we recommend an upgrade to MySQL 5.0.33 source or the expected 5.0.35 binary and source release or 5.1.14. For our paying customers who enjoy the Extended Lifetime Support, we will offer binary builds also to the older releases, in MySQL 4.0.29 and MySQL 4.1.23 respectively. To subscribe to MySQL Enterprise, please visit http://www.mysql.com/enterprise/ .

----
[10 Feb 2007 5:50] Heng Sun
It is sadly "funny" to see this bug report. Our company purchased MySQL Silver support more than two years. We used MySQL 4.x server on a 32-CPU SGI Linux (first Redhat then SuSE) 64-bit machine. Almost from day one (or month one) we started to use MySQL seriously (well before this bug was submitted), we found this problem. We created a detailed test cases and results and sent to MySQL support. But we were kept on being told there were some configuration problem etc, until some senior architect at MySQL told us no support on 32-CPU machine about half year ago. We then rushed to migrate all data to other DB system.

Our test cases showed that the time spent on query only (no write) is almost a linear function of the number of concurrent queries. And it is much worse than a linear function if more then 7 or 8 concurrent queries are performed. At extreme, MySQL server was basic "choked" if over ten concurrent queries were run (results seemed never returned). This means, we were penalized for having multiple CPUs (insteading of MySQL taking advantage of the 32 CPUs). The sad part of story is that we were isolated from this bug report (and MySQL technicians have visited use quite a few times trying to address the issuer).

I myself love MySQL very much. From this report, it seems 5.1 could address the problem. I would love to give it a try. But it is certainly not something fun to migrate huge amount of data back to MySQL and run some old program to test. So if someone has more updated info (like testing on a more than 4-CPU machine), please let me know. Thanks.
[18 Feb 2007 0:12] James Day
MySQL 4.0.29 has been released with binaries for Enterprise customers and contains this improvement.

MySQL 4.1.23 is expected in about 4 weeks, subject to change, and will contain this improvement, with the usual binary for Enterprise customers.

Heng Sun, I suggest testing it but not making a big migration. There are many other places where concurrency can be a problem and 32 cores is too high to say that it has been well tuned for it today. We're still working to improve the server, both the MySQL and Innobase parts. We would be interested in the results you get with your applications but do expect it to take time to improve them.
[27 Feb 2007 18:37] Pierre POMES
Hi all,

Since this bug is now fixed in version >= 5.0.30, is it recommended to re-enable hyperthreading ? 

Many thanks,
Pierre
[27 Feb 2007 22:01] James Day
The next scheduled 4.1 build has been rescheduled to mid summer. Those with Enterprise, Network or other support contracts that provide hotfix builds as part of the package should open a support issue or reply to their existing support issue if they don't want to wait for the next scheduled build.

Pierre, try it and see. Hyperthreading gain or loss has always depended on the particular workload so I can't say if it will be good or bad for you.

James Day, Support Engineer, MySQL AB
[13 Mar 2007 7:22] James Day
MySQL Community Server 5.0.37 has now been released and is the first Community binary built by MySQL to include this change.

James Day, Support Engineer, MySQL AB
[10 May 2007 22:57] James Day
Summary of MySQL releases with this improvement.

Enterprise/Network/other Support customers:

6.0 Falcon preview: now. This is early beta and not for production use.
5.1: 5.1.14 and later. Beta.
5.0: 5.0.30 and later.
4.1: Expected summer 2007 in 4.1.23. Subject to change. BK source already includes it. Customers whose contracts include hotfix builds can request a build.
4.0: 4.0.30 and later. See historical releases on download page.

Community users:

6.0 Falcon preview: now. This is early beta and not for production use.
5.1: 5.1.14 and later, source and binary. Beta.
5.0: Binary 5.0.37 and later, source 5.0.33 and later.
4.1: Unlikely, according to MySQL Lifecycle Policy. Source: see BK. Unstable developer daily snapshots are available at http://downloads.mysql.com/snapshots.php but are not recommended for production use.
4.0: MySQL no longer hosts Community 4.0 downloads. No Community releases included this change. Unstable developer daily snapshots are available at http://downloads.mysql.com/snapshots.php but are not recommended for production use.

In addition, source is normally available in BitKeeper (BK) and assorted Linux operating system distributions are likely to include this.

James Day, Support Engineer, MySQL AB
[10 May 2007 23:15] Dathan Pattishall
Where is the tag for 4.1.23? I see 4.1.22 it's been in bitkeeper  for 7 months but 4.1.23 doesn't exist. Here is where I am looking. 

http://mysql.bkbits.net:8080/mysql-4.1/?PAGE=tags

Also I searched bkbits.net for this bug num  with this query string

http://mysql.bkbits.net:8080/mysql-4.1/?PAGE=search&EXPR=15815&SEARCH=file+comments

nothing that leads me to the 4.1.23 branch
[11 May 2007 0:07] Dathan Pattishall
head version of bkbits is 4.1.23 just not blessed yet, although it hasn't had major changes for quite some time.
[13 Jul 2007 21:06] Heng Sun
Script to test MySQL performance on a multiple CPU server

Attachment: test_parallel.tcl (application/octet-stream, text), 4.73 KiB.

[13 Jul 2007 21:07] Heng Sun
Table structures used in test multiple CPUs

Attachment: tables.sql (text/plain), 1.51 KiB.

[13 Jul 2007 21:27] Heng Sun
We got time to come back to this issue. I do feel this bug is addressed, even beyond 4 CPUs. We have 32-CPU Itenium 2 with SuSE Linux 9 installed. Along with the existing MySQL 4.1.13 installed, another MySQL 5.0.43 (Community Server) is installed.

The test script test_parallel.tcl is uploaded. It is written in Tcl. It runs 3 tests. For each test, it runs a series of cases with multiple client queries running at the same time. 

We only use two tables to test. One has 31 million rows. The other has 84 million rows. The query returns 300K rows.

Here is the summary of a result:

      Parallels         Test 1         Test 2         Test 3
              1     1.18 (min)     0.92 (min)     0.95 (min)
              2     1.18 (min)     0.97 (min)     1.23 (min)
              3     1.28 (min)     1.25 (min)     1.30 (min)
              4     1.60 (min)     1.47 (min)     1.55 (min)
              5     1.95 (min)     1.77 (min)     1.82 (min)
              6     2.23 (min)     1.97 (min)     2.13 (min)
              7     2.37 (min)     2.33 (min)     2.35 (min)
              8     2.78 (min)     2.82 (min)     2.83 (min)
              9     3.15 (min)     3.17 (min)     3.17 (min)

From these numbers, it appears that the fix also goes to machine with more than 4 CPUs. I remember that a similar test I performed one or two years ago had much worse results. At that test, the time spent is an exponential function of the number of parallel queries. Indeed, once there are about 4 or 5 queries running at the same time, the server seems in a deadlock mode.

We'll have more extensive tests.
[15 Jul 2007 12:06] Peter Zaitsev
Heng,

So you're saying you have 32 CPUs on this box  ?   This means if you run this benchmark on this box exclusively in the perfect word (100% scalability) should should have seen same query execution time up to 32 concurrent connections. 

Of course this does not happen because there is no perfect scalability and memory bus can well be bottleneneck etc. 

But still it would be interesting to see the results for your test up to 32 threads (probably with some step)

Now with 9 threads and 32 CPUs you have some 3 times worse performance per thread which is not the perfect scaling for sure
[16 Jul 2007 14:39] Heng Sun
Peter,

I would ask the same question myself. So I did further tests over the weekend, when the machine was not heavily used and it was OK for one program to use up most CPUs. Here are the results. I used a query with less rows returns than the previous one. For running from 1 up to 40 parallel queries, I got

      Parallels         Test 1         Test 2         Test 3
              1     0.58 (min)     0.60 (min)     0.60 (min)
              2     0.67 (min)     0.82 (min)     0.65 (min)
              3     0.88 (min)     0.88 (min)     0.90 (min)
              4     1.07 (min)     1.03 (min)     1.08 (min)
              5     1.30 (min)     1.28 (min)     1.27 (min)
              6     1.50 (min)     1.43 (min)     1.48 (min)
              7     1.70 (min)     1.58 (min)     1.68 (min)
              8     1.82 (min)     1.97 (min)     1.88 (min)
              9     2.22 (min)     2.05 (min)     2.12 (min)
             10     2.45 (min)     2.23 (min)     2.30 (min)
             11     2.63 (min)     2.52 (min)     2.55 (min)
             12     2.65 (min)     2.65 (min)     2.75 (min)
             13     2.98 (min)     2.93 (min)     3.02 (min)
             14     3.23 (min)     3.17 (min)     3.12 (min)
             15     3.45 (min)     3.37 (min)     3.50 (min)
             16     3.68 (min)     3.70 (min)     3.60 (min)
             17     3.85 (min)     3.88 (min)     3.85 (min)
             18     4.22 (min)     4.15 (min)     4.08 (min)
             19     4.37 (min)     4.40 (min)     4.42 (min)
             20     4.58 (min)     4.55 (min)     4.72 (min)
             21     4.70 (min)     4.88 (min)     4.77 (min)
             22     4.95 (min)     5.08 (min)     4.98 (min)
             23     5.17 (min)     5.27 (min)     5.30 (min)
             24     5.48 (min)     5.48 (min)     5.48 (min)
             25     5.63 (min)     5.67 (min)     5.70 (min)
             26     5.80 (min)     5.97 (min)     5.95 (min)
             27     5.98 (min)     6.07 (min)     6.15 (min)
             28     6.40 (min)     6.27 (min)     6.22 (min)
             29     6.58 (min)     6.52 (min)     6.52 (min)
             30     6.83 (min)     6.82 (min)     6.83 (min)
             31     7.02 (min)     7.18 (min)     7.00 (min)
             32     7.30 (min)     7.32 (min)     7.17 (min)
             33     7.48 (min)     7.48 (min)     7.35 (min)
             34     7.77 (min)     7.53 (min)     7.65 (min)
             35     7.83 (min)     7.90 (min)     8.07 (min)
             36     8.08 (min)     8.18 (min)     8.25 (min)
             37     8.35 (min)     8.33 (min)     8.52 (min)
             38     8.47 (min)     8.58 (min)     8.63 (min)
             39     8.78 (min)     8.80 (min)     8.68 (min)
             40     9.08 (min)     8.93 (min)     9.10 (min)

Put the data in a spreadsheet and you will see an almost linear function, even for runs beyond 32 queries in parallel. This would lead to further question: What will we see if we further "stress"? Here are the results which 10x+1 paralle queries:

      Parallels         Test 1         Test 2         Test 3
              1     0.58 (min)     0.58 (min)     0.55 (min)
             11     2.63 (min)     2.63 (min)     2.60 (min)
             21     4.82 (min)     4.80 (min)     4.72 (min)
             31     7.05 (min)     6.87 (min)     7.02 (min)
             41     9.18 (min)     9.17 (min)     9.13 (min)
             51    11.28 (min)    11.33 (min)    11.33 (min)
             61    13.65 (min)    13.85 (min)    13.60 (min)
             71    16.20 (min)    16.00 (min)    16.30 (min)
             81    18.45 (min)    18.50 (min)    18.30 (min)
             91    20.70 (min)    20.62 (min)    20.58 (min)
            101    22.95 (min)    23.07 (min)    22.92 (min)
            111    23.02 (min)    23.00 (min)    23.18 (min)
            121    23.08 (min)    23.13 (min)    23.42 (min)
            131    22.90 (min)    23.28 (min)    22.98 (min)
            141    22.83 (min)    23.32 (min)    23.07 (min)
            151    22.97 (min)    23.10 (min)    23.32 (min)

We have to discard all results start from parallels=111 since I forgot to lift up the default max MySQL client connection number. My logs show all runs since 111 were not completed successfully. The almost constant run time since 101 is actually consistent with the fact that all query processes beyond 101-st got connection refused.

From the data, we see the obvious linear pattern. The positive y-axis intercept means there is a common overhead for all queries. Then each more query adds about 0.22 min time.

From these data, we can at least conclude that one bug is fix: The processing time is no worse than a linear function. As indicated in this bug report, it was not the case before. Also as I mentioned in this bug report (and filed to MySQL support too), older version of MySQL processing time was an exponential function of number of parallel queries.

I don't think from these data, we can conclude that MySQL does not support parallel processing. My "wild" explanation is as follows. The test data we loaded are all in the same physical disk. And the queries are performed on the same set of rows of data. We use the default MySQL lock seeting for InnoDB. The query result cache is disabled. So each query has to access the same hard disk. If hard disk is the bottleneck, the 0.22 min that eacy query adds to the total processing time is the hard disk I/O. The y-axis intercept of 0.16 min (the common overhead) is the CPU processing time.

I know nothing about MySQL internals. So the above is just my guess. To further understand, more tests including laying data into different disks, querying different part of table, and changing the default locking (to row locking for example), should be performed. And, so far we just talk about plain query -- no sorting, no group by, no update, etc.

Before this test, I had three wishes for MySQL parallel computing in mind:

Step 1: Fix the critical bug that makes processing time even worse than linear function of number of queries. (Seems fixed now)

Step 2: Parallel processing queries. So processing time should be a function must better than a linear one if lock setting and data allocation are proper.

Step 3: Even for a single big query, the server should slit the query into several CPUs. So for a single big query, we should expect a better time when more CPUs are visible to the query. Previously I learned from MySQL that this was not even targeted in theory. We know this is achieved by those commercial database vendors like MicroSoft SQL Server. Of course, MySQL may strategically emphasize on other routes like clustered db servers.

That concludes out first round test. We may or may not proceed to further test depending on our decision whether to use MySQL.
[16 Jul 2007 17:58] Peter Zaitsev
My question is when if you're really measuring CPU scalability not something else here ? 

If your workload would be heavily CPU bound you should have saturated CPUs closely to 32 threads.    But how you can go to 100 threads  scaling linearly with CPU bound load and 32 CPUs I do not know. 

Did you check the CPU load with single thread benchmark for example ? 

Anyway I agree with you some bug was fixed in a sense the same test showed much worse results before.
[16 Jul 2007 20:17] Heng Sun
Yes, I forgot to mention about CPU usage. I did monitor from time to time. Indeed, when more queries ran in parallel, I saw CPU usage from less than 100% to 800%. 'ps -Lmf' showed the number of new threads coming out of mysqld equals the number of concurrent queries. 

I never saw the CPU usage over 800%. As I mentioned, the test used a query that fetching the same data from the two tables sitting in the same hard disk. The disk IO might be the bottleneck to make full CPU usage impossible. This is why (I think) I saw over 100 threads out of mysqld but CPU usage is no more than 800% (in 'Irix mode') Or in solaris mode, the CPU usage is limited by 25%. 

But since you asked this question, I suddently realized this 25% might be a thredshold set for each process. I asked our admin and the answer was no limit was ever set except possible by default. There got to be something about this mystery 25%.
[16 Jul 2007 20:35] Heng Sun
One small correction: If I run only one query, the mysqld CPU usage is 100% (or 101% due to small activities from other mysqld threads).
[24 Jul 2007 15:23] Heng Sun
Hi, kishorekumar,

Who do you ask to execute your queries? If I am included, please give the SQL 'create table' for the table structure. Thanks.
[13 Sep 2007 12:59] Jonatas Cruz
Was this patch included in the MySql 5.0.41 ?

Regards
Jonatas
[25 Jan 2008 12:38] Bugs System
Pushed into 6.0.5-alpha
[28 Jan 2008 6:50] Bugs System
Pushed into 5.1.24-rc
[28 Jan 2008 6:51] Bugs System
Pushed into 5.0.56
[28 Jan 2008 6:52] Bugs System
Pushed into 4.1.24
[28 Jan 2008 9:56] Peter Zaitsev
What is happening with this Bug number ?
It seems to be reused multiple times - it was in "Closed" state for a while and now I see something else pushed in bunch of trees. 

Any idea what these new patches are ?
[28 Jan 2008 20:18] Timothy Smith
Peter & others,

Sorry for the noise from the bugs system.  The only real change is that the patch is now in the 4.1 tree as well.  The 4.1 patch had been kept in a separate clone for a long time, and had never made it to the main 4.1 tree.  Now it's there.

The others are just "null" merges up the line; they are just noise.

Oddly enough, this *has* been in the 4.0 tree for a long time; just the 4.1 tree was overlooked.
[29 Jan 2008 8:28] Kevin Burton
NICE....... I'm glad this patch made it into the 4.1.x tree.  I'm def going to put this into production when it's ready.
[14 Mar 2008 9:34] Domas Mituzas
Note: this patch might be the cause of +10% memory consumption tax on top of buffer pool.
[5 May 2010 15:20] Bugs System
Pushed into 5.1.47 (revid:joro@sun.com-20100505145753-ivlt4hclbrjy8eye) (version source revid:vasil.dimov@oracle.com-20100331130613-8ja7n0vh36a80457) (merge vers: 5.1.46) (pib:16)
[6 May 2010 1:45] Paul DuBois
Push resulted from incorporation of InnoDB tree. No changes pertinent to this bug. Re-closing.
[28 May 2010 5:50] Bugs System
Pushed into mysql-next-mr (revid:alik@sun.com-20100524190136-egaq7e8zgkwb9aqi) (version source revid:vasil.dimov@oracle.com-20100331130613-8ja7n0vh36a80457) (pib:16)
[28 May 2010 6:19] Bugs System
Pushed into 6.0.14-alpha (revid:alik@sun.com-20100524190941-nuudpx60if25wsvx) (version source revid:vasil.dimov@oracle.com-20100331130613-8ja7n0vh36a80457) (merge vers: 5.1.46) (pib:16)
[28 May 2010 6:47] Bugs System
Pushed into 5.5.5-m3 (revid:alik@sun.com-20100524185725-c8k5q7v60i5nix3t) (version source revid:vasil.dimov@oracle.com-20100331130613-8ja7n0vh36a80457) (merge vers: 5.1.46) (pib:16)
[29 May 2010 15:27] Paul DuBois
Push resulted from incorporation of InnoDB tree. No changes pertinent to this bug.
Re-closing.
[17 Jun 2010 11:51] Bugs System
Pushed into 5.1.47-ndb-7.0.16 (revid:martin.skold@mysql.com-20100617114014-bva0dy24yyd67697) (version source revid:vasil.dimov@oracle.com-20100331130613-8ja7n0vh36a80457) (merge vers: 5.1.46) (pib:16)
[17 Jun 2010 12:28] Bugs System
Pushed into 5.1.47-ndb-6.2.19 (revid:martin.skold@mysql.com-20100617115448-idrbic6gbki37h1c) (version source revid:vasil.dimov@oracle.com-20100331130613-8ja7n0vh36a80457) (merge vers: 5.1.46) (pib:16)
[17 Jun 2010 13:16] Bugs System
Pushed into 5.1.47-ndb-6.3.35 (revid:martin.skold@mysql.com-20100617114611-61aqbb52j752y116) (version source revid:vasil.dimov@oracle.com-20100331130613-8ja7n0vh36a80457) (merge vers: 5.1.46) (pib:16)