MySQL Bugs: #110162: Improve vio read performance by introducing short busy poll before sleep

Bug #110162	Improve vio read performance by introducing short busy poll before sleep
Submitted:	22 Feb 2023 7:06	Modified:	23 Feb 2023 1:51
Reporter:	Cai Yibo (OCA)	Email Updates:
Status:	Not a Bug	Impact on me:	None
Category:	MySQL Server	Severity:	S5 (Performance)
Version:	8.0	OS:	Linux
Assigned to:		CPU Architecture:	ARM
Tags:	performance, tcpip, vio

Description:
https://github.com/mysql/mysql-server/blob/mysql-8.0.32/vio/viosocket.cc#L135

For VIO tcp socket reading, it first does a non-block read. If the read might block (EAGAIN), it immediately calls ppoll (vio_socket_io_wait -> vio_io_wait -> ppoll) to sleep and wait for incoming packets.

Below pseudo code illustrates the idea, though greatly simplified.

vio_read(...) {
    while (recv_non_block(sockfd) == -1) {
        if (errno != EAGAIN) return failed;
        ppoll((sockfd, PULLIN), timeout);
    }
    return success;
}

This pattern may be sub-optimal when incoming requests are intensive. The ppoll syscall introduces considerable syscall and context switch overhead, as can be seen from on-cpu and off-cpu flame graphs.

Instead of abandoning CPU immediately when there's no data, we can spin in a busy loop for a short time to peek data readiness repeatly, and hope the data will come soon to avoid the ppoll syscall and context switch.
This is similar to rwlock, if it's already locked, the lock() function spins to acquire the lock repeatly for some time before sleep.

For my test case (80 Arm Neoverse-N1 cores, sysbench write-only workload, 16 theads), I saw stable performance improvement after applying this trick.
- TPS increases about 10%
- Latency decreases about 10%

Of course, the performance depends heavily on system configuration and use cases. The delay and loop counts should be configurable, just like rwlock parameters innodb_spin_wait_delay, innodb_sync_spin_loops, etc.

Would like to hear community comments.

How to repeat:
Test env:
- 80 core Arm Neoverse-N1 server
- mysql 8.0.32
- sysbench 1.0.20
- gcc-10.3, aarch64, cmake build type = relwithdebinfo
- linux kernel 5.14

Test case:
- sysbench write-only workload, 16 theads

add version

Hi Mr. Yibo,

Thank you for your performance feature request.

We have to analyse much more carefully, which includes our performance department. Introducing any additional code could decrease the performance.

Hence, you need to send us your code that would show much faster performance on both Intel and ARM platforms.

We do NOT like adding any new configuration variables, so that request will not be accepted.

But, seeing your improve  code and EXACT performance improvement is obligatory demand from those that want to introduce changes.

When you answer us next time, please make sure, that we see attached  your code that performs this functionality and measured performance improvements. For the measurement we accept only mysqlslap and sysbench.

Let us also warn you that we are NOT prone to change the code that works, so please make sure you send everything demanded. Do note that small improvements will (most likely) lead to the closure of your report.  We shall check your findings and your code, when we get both.

Last, but not least, introduction of any spinlock will lead to rejection of your proposal, so we suggest checking out those as well.

We are patiently waiting on your FUlLL source code diff and all benchmark measurements.

Another small request from us.

We would like to see sysbench results with 2, 8 , 16, 32 and 64 concurrent connection / threads.

Sorry for this addition ........

Hi Mr. Yibo,

Sorry, but we discovered that you have sent us some unknown code.

Our current release, 8.0.32, has already everything that you asked for in this report.

Hence, this is not a bug !!!!!

Sorry, but I didn't send any code related to this bug report...

I did submitted a patch to fix rwlock memory order bug years ago [1], and some other trivial ones. This bug report is from an inhouse experiment this week, and I want to get community comments early before writing a full patch that doesn't meet expectations.

Anyway, greatly thanks for the detailed explanation about the requirements to accept such an improvement. I totally agree we should be extremely careful about changing existing code, and solid performance evaluation reports must be provided for various cases and platforms.

I'm okay we close this bug report. My current experiment code doesn't meet the requirement. I do believe there's performance potential in vio socket code and will investigate deeper. Hopefully I can come up with a robust implementation for community review, together with benchmark reports.

Thanks again.

[1] https://bugs.mysql.com/bug.php?id=94699

Hi Mr. Yibo,

Your first entry into this newly created bug is the following one:

-----------------------------------------------------------------

[22 Feb 7:06] Cai Yibo
Description:
https://github.com/mysql/mysql-server/blob/mysql-8.0.32/vio/viosocket.cc#L135

For VIO tcp socket reading, it first does a non-block read. If the read might block (EAGAIN), it immediately calls ppoll (vio_socket_io_wait -> vio_io_wait -> ppoll) to sleep and wait for incoming packets.

Below pseudo code illustrates the idea, though greatly simplified.

vio_read(...) {
    while (recv_non_block(sockfd) == -1) {
        if (errno != EAGAIN) return failed;
        ppoll((sockfd, PULLIN), timeout);
    }
    return success;
}

This pattern may be sub-optimal when incoming requests are intensive. The ppoll syscall introduces considerable syscall and context switch overhead, as can be seen from on-cpu and off-cpu flame graphs.

Instead of abandoning CPU immediately when there's no data, we can spin in a busy loop for a short time to peek data readiness repeatly, and hope the data will come soon to avoid the ppoll syscall and context switch.
This is similar to rwlock, if it's already locked, the lock() function spins to acquire the lock repeatly for some time before sleep.

For my test case (80 Arm Neoverse-N1 cores, sysbench write-only workload, 16 theads), I saw stable performance improvement after applying this trick.
- TPS increases about 10%
- Latency decreases about 10%

Of course, the performance depends heavily on system configuration and use cases. The delay and loop counts should be configurable, just like rwlock parameters innodb_spin_wait_delay, innodb_sync_spin_loops, etc.

Would like to hear community comments.

How to repeat:
Test env:
- 80 core Arm Neoverse-N1 server
- mysql 8.0.32
- sysbench 1.0.20
- gcc-10.3, aarch64, cmake build type = relwithdebinfo
- linux kernel 5.14

Test case:
- sysbench write-only workload, 16 theads

-------------------------------------------------------------------------

Can't you see it ????