MySQL Bugs: #113742: the container' file system changed to read-only

Bug #113742	the container' file system changed to read-only
Submitted:	24 Jan 2024 13:37	Modified:	24 Jan 2024 19:50
Reporter:	Bing Ma (OCA)	Email Updates:
Status:	Can't repeat	Impact on me:	None
Category:	MySQL Operator	Severity:	S1 (Critical)
Version:	8.2.0-2.1.1	OS:	Any
Assigned to:	MySQL Verification Team	CPU Architecture:	Any

Description:
After running for a period of time, the file system of the mysql container becomes read-only

bash-4.4$ cd /
bash-4.4$ touch 123.txt
touch: cannot touch '123.txt': Read-only file system
bash-4.4$ cd /var/lib
bash-4.4$ touch 123.txt
touch: cannot touch '123.txt': Read-only file system
bash-4.4$

How to repeat:
After running for a period of time

in /livenessprobe.sh,you write to /fail-counter

bash-4.4$ cat /livenessprobe.sh
#!/bin/bash
# Copyright (c) 2020, 2021, Oracle and/or its affiliates.

# Insert 1 success every this amount of failures
# (assumes successThreshold is > 1)
max_failures_during_progress=$1

# Ping the server to see if it's up
mysqladmin -umysqlhealthchecker ping
# If it's up, we succeed
if [ $? -eq 0 ]; then
  exit 0
fi

if [ -z $max_failures_during_progress ]; then
  exit 1
fi

# If the init/startup/InnoDB recovery is still ongoing, we're
# not succeeded nor failed yet, so keep failing and getting time
# extensions until it succeeds.
# We currently rely on the server to exit/abort if the init/startup fails,
# but ideally there would be a way to check whether the server is
# still making progress and not just stuck waiting on a frozen networked
# volume, for example.

if [ -f /fail-counter ]; then
  fail_count=$(($(cat /fail-counter) + 1))
else
  fail_count=1
fi

if [ $fail_count -gt $max_failures_during_progress ]; then
  # Report success to reset the failure counter upstream and get
  # a time extension
  rm -f /fail-counter
  exit 0
else
  # Update the failure counter and fail out
  echo $fail_count > /fail-counter
  exit 1
fi

Hi,

Thank you for using MySQL Server. I cannot reproduce the problem.