MySQL Bugs: #81229: Client performance trashed due to threads woken up too early

Bug #81229	Client performance trashed due to threads woken up too early
Submitted:	29 Apr 2016 7:50	Modified:	10 May 2016 10:07
Reporter:	Ole John Aske	Email Updates:
Status:	Closed	Impact on me:	None
Category:	MySQL Cluster: NDB API	Severity:	S3 (Non-critical)
Version:	7.3	OS:	Any
Assigned to:		CPU Architecture:	Any

Description:
When the API clients wait for replies after having sent a request (or signal),
they will wait in a do_poll() call with a maximum wait time specified.
After this max wait has expired, they will be woken up, regardless of
whether they got a reply or not. As any other conditional waits, there 
may also be spurious wake ups in addition to this. In any cases the
client had to check for the proper reply being received, and continue
waiting if required.

Throughout our API client code, there has been a common pattern of
doing such poll wait with a max wait time of 10ms. If the client 
specified a longer timeout, it was broken up in several 10ms chunks.
The reason has likely been that 10ms was believed to be an eternity
compared to how long a datanode request normally takes. Which for
most use cases normally is true.

However, when overloading the system sufficiently, it is possible to
enter a state where requests starts taking more than 10ms. We will
then start timing out on the do_poll() waits, to immediately being
put back into a new wait. This adds overhead to the already 
overloaded system, which trash the system performance further.

How to repeat:
Create a API client with several thousand of threads sending simple operations to the datanodes.

Suggested fix:
Wait the specified amount of milliseconds without doing it in 10ms chunks when not needed for other reasons.

Documented fix as follows in the NDB 7.5.2 changelog:

    A performance problem was found in an internal polling method
    do_poll() where the polling client did not check whether it had
    itself been woken up before completing the poll. Subsequent
    analysis showed that it is sufficient that only some clients in
    the polling queue receive data. do_poll() can then signal these
    clients and give up its polling rights, even if the maximum
    specified wait time (10 ms) has not expired. 

    This change allows do_poll() to continue polling until either 
    the maximum specified wait time has expired, or the polling 
    client itself has been woken up (by receiving what it was 
    waiting for). This avoids unnecessary thread switches between 
    client threads and thus reduces the associated overhead by as 
    much as 10% in API clients, resulting in a significant 
    performance improvement when client threads perform their own 
    polling.

Closed.