Bug #99966 Switching to use NUMA-SMART Counter for statistical counters
Submitted: 24 Jun 2020 8:12 Modified: 30 Jun 2020 2:07
Reporter: Krunal Bauskar Email Updates:
Status: Verified Impact on me:
Category:MySQL Server: InnoDB storage engine Severity:S5 (Performance)
Version:8.0.20 OS:Any
Assigned to: CPU Architecture:ARM
Tags: arm, numa, PowerPC, x86

[24 Jun 2020 8:12] Krunal Bauskar
- MySQL recently introduced a Shard Counter based framework. This framework expects the caller to pass the shard-id/slot.
- As per my understanding, the framework is based on the idea that by having N shards, contention will reduce. While it could hold true core-affinity is being lost. This will cause a shard to move around depending on which core is trying to update the said page.
- This movement, especially on NUMA node, introduces serious performance drop.
- Also, on the workload that accesses a limited number of pages, contention arising from page-id use will further affect performance.
- Based on the study and experiment patch tend to propose a self-contained     ib_atomic_counter_t (on the line of ib_counter_t) that does consider the core-affinity.
- Patch has proved to show improvement up to 55% on NUMA enabled machine.

How to repeat:
* Apply the patch that I will submit below and try.

Suggested fix:
* Performance fix.
[24 Jun 2020 8:13] Krunal Bauskar
BMS-ARM Pattern-1 benchmarking result

Attachment: bms.arm.pattern1.png (image/png, text), 70.29 KiB.

[24 Jun 2020 8:14] Krunal Bauskar
VM-x86 Pattern 1 results

Attachment: vm.x86.pattern1.png (image/png, text), 62.27 KiB.

[24 Jun 2020 8:14] Krunal Bauskar
BMS-ARM pattern2 results

Attachment: bms.arm.pattern2.png (image/png, text), 69.31 KiB.

[24 Jun 2020 8:16] Krunal Bauskar
I have added 3 benchmarking results
1. BMS-arm-pattern-1 (zipfian)
2. vm-x86-pattern-1 (zipfian)
3. BMS-arm-pattern-2 (uniform)

Machine details:
BMS: 128 cores, ARM (Kunpeng), with 4 NUMA nodes. 100tables*5m all in memory
VM: 24 vCPU, ARM (x86 Xeon(R) Gold 6266C CPU @ 3.00GHz) with 2 NUMA nodes. 64t*1.5m in memory.
[24 Jun 2020 12:10] MySQL Verification Team
Hi Mr. Bauskar,

Thank you for your performance improvement report.

I have analysed all of the attachments and I have to agree with you.

Verified as reported.
[25 Jun 2020 1:40] Krunal Bauskar
I see arch was updated to ARM.

Please note, the said patch/idea is not ARM specific it is meant to help NUMA which is also found in x86. I even show-cased result for x86.

Just that my x86 machine it not that strong as ARM.

Can we switch it back to "Any"
[25 Jun 2020 11:49] MySQL Verification Team
Sure. Done.
[25 Jun 2020 12:23] Krunal Bauskar
Thanks. I appreciate your quick response.
[30 Jun 2020 2:07] Krunal Bauskar
Not sure why the CPU-arch was reset to ARM. Unfortunately, happening for 2nd time.

I would appreciate it if someone can leave a note while moving things explaining why they feel this bug is ARM specific issue given NUMA has nothing to do with ARM only.
[30 Jun 2020 12:45] MySQL Verification Team

The CPU architecture has been changed by our Development, but I am notifying them that this is not ARM-only bug report.