Bug #106697 Missing sync can lead to running out of memory in machine

• Submitted: 10 Mar 2022 15:08
• Modified: 13 May 2022 17:01
• Reporter: Mikael Ronström
• Status: Closed
• Category: MySQL Cluster: Cluster (NDB) storage engine
• Severity: S3 (Non-critical)
• Version: 8.0.28
• OS: Linux
• CPU Architecture: Any

[10 Mar 2022 15:08] Mikael Ronström

Description:
When initialising REDO logs, Tablespace files and UNDO log files we use
the flag OM_INIT that will invoke writes in a loop of the pages in the file.
We prepare this by setting the autosync rate to 1 MByte. However we
will never check this by calling sync_on_write. Thus if the file system cannot
keep up with the write speed, we can use massive amounts of memory in
Linux leading to Out Of Memory situations.

Fixed by adding a call to sync_on_write in the write loop.
Also increased the 1 MByte to 16 MByte.

How to repeat:
Need a file system that can't keep up with the writes and
creating some large tablespace file or logfile group.

Suggested fix:
Ensure sync_on_write is called in loop.

[10 Mar 2022 17:06] MySQL Verification Team

Hi,

Thanks for the report Mikael,

all best
Bogdan

[13 May 2022 17:01] Jon Stephens

DOcumented fix as follows in the NDB 8.0.30 changelog:

    When initializing a file, NDBFS enabled autosync but never
    called do_sync_after_write() (then called sync_on_write()), so
    that the file was never synchronized to disk until it was saved.
    This meant that, for a system whose network disk was stalled for
    some time, the file could use up system memory on buffered file
    data.

    We fix this by calling do_sync_after_write() each time NDBFS
    writes to a file.

Closed.