| Bug #116539 | MySQL 8.0/8.4 performs worse on RHEL9 than on RHEL8 | ||
|---|---|---|---|
| Submitted: | 4 Nov 2024 14:15 | Modified: | 6 Nov 2024 9:22 |
| Reporter: | Truls Bergskaug | Email Updates: | |
| Status: | Not a Bug | Impact on me: | |
| Category: | MySQL Server | Severity: | S3 (Non-critical) |
| Version: | 8.0, 8.4 | OS: | Red Hat (RHEL9) |
| Assigned to: | CPU Architecture: | x86 (Any) | |
[4 Nov 2024 14:29]
MySQL Verification Team
Hi Mr. Bergskaug, Thank you for your bug report. You have provided the following sentence: " Using MySQL 8.0 and 8.4 (different patches) performs worse on RHEL9 than on RHEL8 using TCP/IP connection. Local connect (localhost) seems to work fine. " This proves that a problem is in TCP/IP implementations on different versions of the Red Hat Linux. The problem is in TCP/IP. The fact that differences in 8.4 are smaller is due to some changes in the client-server protocol in 8.4. Code is simplified, since there is no more native password, a number of system settings have been disabled and so on and so forth ...... You can read all about it in 8.4 Reference Manual. Not a bug.
[4 Nov 2024 15:37]
Terje Røsten
RHEL9 is using OpenSSL 3.0, can you try to repeat with TLS/SSL disabled on RHEL9?
[4 Nov 2024 15:45]
MySQL Verification Team
Thank you, Terje ....
[6 Nov 2024 9:22]
Truls Bergskaug
We have now tested with or without ssl enabled, thanks to hint from Terje Røsten.
With SSL enabled:
rhel9, MySQL 8.0.36:
[root@m8r9-bserver1.lab ~]# mysqlslap -a -h 10.144.8.88 -i 100 -c 100
Benchmark
Average number of seconds to run all queries: 3.628 seconds
Minimum number of seconds to run all queries: 3.314 seconds
Maximum number of seconds to run all queries: 4.099 seconds
Number of clients running queries: 100
Average number of queries per client: 0
Without ssl (user auth with 'mysql_native_password'.
[root@m8r9-bserver1.lab ~]# mysqlslap -a -u mysqlslap -p -h 10.144.8.88 -i 100 -c 100
Enter password:
Benchmark
Average number of seconds to run all queries: 0.162 seconds
Minimum number of seconds to run all queries: 0.129 seconds
Maximum number of seconds to run all queries: 0.356 seconds
Number of clients running queries: 100
Average number of queries per client: 0
Skipping SSL seems to solve the performance-issue. It seems that MySQL 8.4 not allows to skip ssl, so using MySQL 8.4 may show very different performance using socket and TCP connections when using RHEL9
[6 Nov 2024 11:07]
MySQL Verification Team
Thank you Mr. Bergskaug, For confirming the cause of the performance degradation. TLS/SSL libraries shipped with OS seem to get more and more complex.
Description: Using MySQL 8.0 and 8.4 (different patches) performs worse on RHEL9 than on RHEL8 using TCP/IP connection. Local connect (localhost) seems to work fine. Simple testing, using mysqlslap on different setups shows big difference between versions using RedHat RHEL9 and RHEL8. The testing has been executed on virtual servers (VMWare). The initial discovery (tests not shown here) was on a physical production server with 384G memory, Intel Xeon 6434 (hypthreding on:16 threads), performing worse than a virtual server with lower specs. How to repeat: Test RHEL8 / M8.0.40 (other activity on server) Linux 4.18.0-553.22.1.el8_10.x86_64 [root@ysql8-r8 ~]# mysqlslap -u testuser -p -h localhost -a --concurrency=50 --iterations=10 --verbose Enter password: Benchmark Average number of seconds to run all queries: 0.067 seconds Minimum number of seconds to run all queries: 0.057 seconds Maximum number of seconds to run all queries: 0.118 seconds Number of clients running queries: 50 Average number of queries per client: 0 [root@mysql8-r8 ~]# mysqlslap -u testuser -p -h 127.0.0.1 -a --concurrency=50 --iterations=10 --verbose Enter password: Benchmark Average number of seconds to run all queries: 0.238 seconds Minimum number of seconds to run all queries: 0.204 seconds Maximum number of seconds to run all queries: 0.367 seconds Number of clients running queries: 50 Average number of queries per client: 0 [root@mysql8-r8 ~]# mysqlslap -u testuser -p -h $(uname -n) -a --concurrency=50 --iterations=10 --verbose Enter password: Benchmark Average number of seconds to run all queries: 0.230 seconds Minimum number of seconds to run all queries: 0.192 seconds Maximum number of seconds to run all queries: 0.336 seconds Number of clients running queries: 50 Average number of queries per client: 0 Test RHEL9 / M8.0.35 (no other activity on server) Linux 5.14.0-362.13.1.el9_3.x86_64 [root@mysql8-r9 ~]# mysqlslap -u testuser -p -h localhost -a --concurrency=50 --iterations=10 --verbose Enter password: Benchmark Average number of seconds to run all queries: 0.030 seconds Minimum number of seconds to run all queries: 0.027 seconds Maximum number of seconds to run all queries: 0.035 seconds Number of clients running queries: 50 Average number of queries per client: 0 [root@mysql8-r9 ~]# mysqlslap -u testuser -p -h 127.0.0.1 -a --concurrency=50 --iterations=10 --verbose Enter password: Benchmark Average number of seconds to run all queries: 1.677 seconds Minimum number of seconds to run all queries: 1.645 seconds Maximum number of seconds to run all queries: 1.714 seconds Number of clients running queries: 50 Average number of queries per client: 0 [root@mysql8-r9 ~]# mysqlslap -u testuser -p -h $(uname -n) -a --concurrency=50 --iterations=10 --verbose Enter password: Benchmark Average number of seconds to run all queries: 1.658 seconds Minimum number of seconds to run all queries: 1.637 seconds Maximum number of seconds to run all queries: 1.682 seconds Number of clients running queries: 50 Average number of queries per client: 0 Test RHEL9/M8.4.3 (t) Linux 5.14.0-427.18.1.el9_4.x86_64 [root@mysqllab-rhel91.lab ~]# mysqlslap -u testuser -p -h localhost -a --concurrency=50 --iterations=10 --verbose Enter password: Benchmark Average number of seconds to run all queries: 0.113 seconds Minimum number of seconds to run all queries: 0.095 seconds Maximum number of seconds to run all queries: 0.185 seconds Number of clients running queries: 50 Average number of queries per client: 0 [root@mysqllab-rhel91.lab ~]# mysqlslap -u testuser -p -h bf-mysqllab-rhel91.p1.lab -a --concurrency=50 --iterations=10 --verbose Enter password: Benchmark Average number of seconds to run all queries: 1.517 seconds Minimum number of seconds to run all queries: 1.322 seconds Maximum number of seconds to run all queries: 1.660 seconds Number of clients running queries: 50 Average number of queries per client: 0 --- [root@mysqllab-rhel92.lab ~]# mysqlslap -u testuser -p -h mysqllab-rhel91.p1.lab -a --concurrency=50 --iterations=10 --verbose Enter password: Benchmark Average number of seconds to run all queries: 1.623 seconds Minimum number of seconds to run all queries: 1.285 seconds Maximum number of seconds to run all queries: 2.451 seconds Number of clients running queries: 50 Average number of queries per client: 0 Same tests using MySQL 5.6.51 on RHEL7 and MacBook pro (M1 processor) using MySQL 8.4.3 shows minimal difference using localhost or 127.0.0.1 network interface.