MySQL Bugs: #102463: The table is full - errors at 70% data usage

Bug #102463	The table is full - errors at 70% data usage
Submitted:	3 Feb 2021 8:28	Modified:	3 Feb 2021 10:13
Reporter:	Hendrik Woltersdorf	Email Updates:
Status:	Not a Bug	Impact on me:	None
Category:	MySQL Cluster: Cluster (NDB) storage engine	Severity:	S2 (Serious)
Version:	7.4.27	OS:	Any
Assigned to:	MySQL Verification Team	CPU Architecture:	Any

Description:
Last Friday we got errors like "The table 'GLOBAL_ID' is full" in a production cluster, while "Data usage" showed "70%"

DataMemory=64G
IndexMemory=10G

2021-01-29 09:43:03 11673 [ERROR] /opt/mysql/bin/mysqld: The table 'GLOBAL_ID' is full
2021-01-29 09:43:04 11673 [ERROR] /opt/mysql/bin/mysqld: The table 'XCLOCK' is full

2021-01-29 09:43:00 [MgmtSrvr] INFO     -- Node 10: Data usage is 70%(1481918 32K pages of total 2097152)
2021-01-29 09:43:00 [MgmtSrvr] INFO     -- Node 10: Index usage is 45%(480828 8K pages of total 1049088)
2021-01-29 09:43:54 [MgmtSrvr] INFO     -- Node 11: Data usage is 70%(1481977 32K pages of total 2097152)
2021-01-29 09:43:54 [MgmtSrvr] INFO     -- Node 11: Index usage is 45%(480632 8K pages of total 1049088)
2021-01-29 09:44:01 [MgmtSrvr] INFO     -- Node 10: Data usage is 70%(1482244 32K pages of total 2097152)
2021-01-29 09:44:01 [MgmtSrvr] INFO     -- Node 10: Index usage is 45%(480609 8K pages of total 1049088)

Obviously the reported Data usage of 70% is wrong. Is this a known problem? How can we get the "real" Data usage? 
This was a very nasty surprise.

How to repeat:
No idea.

reduced config and log files

Attachment: ndb_error_report_20210203092302.zip (application/x-zip-compressed, text), 2.85 MiB.

Hi,

This is not a bug. "Table full" can happen for other reasons than "not enough memory", e.g. you can be out of partitions (solve by alter..max_rows), your RAM could be fragmented (solve by rolling restart)...

https://dev.mysql.com/doc/refman/8.0/en/faqs-mysql-cluster.html#faq-cluster-why-error-1114

all best
Bogdan

Among the "full" tables are some of the smallest (very few rows at a time).
That means, max_rows was not the issue. 
RAM fragmentation: May be, but is there any chance, to see that problem coming?

In the meantime we restartet the datanodes and configured more RAM.

Hi,

> RAM fragmentation: May be, but is there any chance, to see that problem coming?

Not that I know of :(

70% is when this problem normally start to happen. Most big users have a policy that if usage hits 70% they take the installation under advisement (the first step is usually rolling restart and the second step is adding more nodes or more RAM). A lot has changed throughout versions too in order to change this behavior so 8.0 is handling this much better. 

Hope this at least helps a bit, MySQL Cluster Support team and MySQL Enterprise tools would surely be more help but you need a subscription for those (IMHO very very very useful subscription, but don't listen to me, ping some of the users that have it and ask how happy they are with it :) ).

Kind regards
Bogdan