Bug #110846 | NDBD node shutdown forced by error 2334 - Job Buffer Full | ||
---|---|---|---|
Submitted: | 27 Apr 2023 13:48 | Modified: | 11 May 2023 17:14 |
Reporter: | Tomasz Cios | Email Updates: | |
Status: | Verified | Impact on me: | |
Category: | MySQL Cluster: Cluster (NDB) storage engine | Severity: | S2 (Serious) |
Version: | 8.0.33 | OS: | Red Hat (8.2) |
Assigned to: | CPU Architecture: | x86 |
[27 Apr 2023 13:48]
Tomasz Cios
[28 Apr 2023 15:53]
Tomasz Cios
One remark: with --opbatch it seems to work fine. I tried with --opbatch=10 and --opbatch=20 - performance is still pretty good on my test environment and ndbd does not brake. I have not tried to find the value of this param when it fails.
[4 May 2023 5:00]
MySQL Verification Team
Hi, I cannot reproduce this and this does not look like a bug but improper configuration of the ndbcluster Can you provide a reproducible test case? Thanks
[4 May 2023 15:23]
MySQL Verification Team
Hi, > Is there a downloadable image of a VM with "properly configured ndbcluster"? No, and it would not work as MySQL Cluster need to be sized properly to your application / way you are using it. There is no "fit all" configuration with MySQL Cluster especially as it is designed to crash when it cannot keep up with tasks rather then slow down (what InnoDB would do). That is also why support and consulting for ndbcluster is on a whole other level compared to enterprise MySQL (InnoDB). > If that is not possible - which improper setting could lead to this error? As you already noted, limiting number of operations per batch with opbatch solved a problem. Check out parameters about max operations configuring data node: https://dev.mysql.com/doc/refman/8.0/en/mysql-cluster-params-ndbd.html and increase to match your needs.
[5 May 2023 0:25]
MySQL Verification Team
Hi, Discussed with colleagues, I do not believe this is a bug but it could be in theory. You are using ndbd (single threaded data node), you would get better results with ndbmtd (multithreaded data node). You have triggers (crash show deferred triggers are involved, ones used with FK's) If you can share your schema I can try to reproduce this behavior and detect if we actually do have a bug or it is just about sizing the cluster. Workaround is as you already noticed - to use smaller transactions. Also, changing triggers from NO ACTION to RESTRICT could help too
[11 May 2023 17:14]
MySQL Verification Team
Hi, Thanks for the data, I managed to reproduce the problem. NDB team will take from now to see if they can point the reason for the crash and fix it. all best