Bug #107928 MySQL 8.0 Replication breaks when tmpdir space is filled
Submitted: 20 Jul 2022 23:09 Modified: 21 Jul 2022 13:27
Reporter: Juan Arruti Email Updates:
Status: Verified Impact on me:
None 
Category:MySQL Server: Replication Severity:S2 (Serious)
Version:8.0 OS:Any
Assigned to: CPU Architecture:Any

[20 Jul 2022 23:09] Juan Arruti
Description:
MySQL 8.0 also breaks replication if Source database tmpdir gets filled. Transaction on Source database is committed, but the Replica is not able to read the binlog file:

slave1 [localhost:21924] {msandbox} ((none)) > select @@version ;
+-----------+
| @@version |
+-----------+
| 8.0.29    |
+-----------+

slave1 [localhost:21924] {msandbox} ((none)) > show replica status \G
...
                Last_IO_Errno: 13114
                Last_IO_Error: Got fatal error 1236 from master when reading data from binary log: 'bogus data in log event; the first event 'mysql-bin.000001' at 8329, the last event read from './mysql-bin.000002' at 16129187, the last byte read from './mysql-bin.000002' at 16129206.'
               Last_SQL_Errno: 13121
               Last_SQL_Error: Relay log read failure: Could not parse relay log event entry. The possible reasons are: the master's binary log is corrupted (you can check this by running 'mysqlbinlog' on the binary log), the slave's relay log is corrupted (you can check this by running 'mysqlbinlog' on the relay log), a network problem, the server was unable to fetch a keyring key required to open an encrypted relay log file, or a bug in the master's or slave's MySQL code. If you want to check the master's binary log or slave's relay log, you will be able to know their names by issuing 'SHOW SLAVE STATUS' on this slave.

How to repeat:
I used the same test case as bug report 86991. 

-- Created a machine with a small tmpdir:

Filesystem      Size  Used Avail Use% Mounted on
/dev/loop1      2.0M   30K  1.8M   2% /mnt/tmp

-- MySQL configuration file:

[mysqld]
tmpdir = /mnt/tmp
binlog_cache_size = 4M
binlog_row_image = minimal
innodb_tmpdir  = /mnt/tmp

-- Commands

USE test;
SOURCE create_random_data.sql
CREATE TABLE `movies_catalog` (
  `movie_id` int(11) unsigned NOT NULL DEFAULT '0',
  `catalog_id` int(11) unsigned NOT NULL DEFAULT '0',
  `movie_is_new_release` tinyint(1) NOT NULL DEFAULT '0',
  PRIMARY KEY (`movie_id`,`catalog_id`),
  KEY `fk_movie_id` (`movie_id`),
  KEY `fk_catalog_id` (`catalog_id`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8 COLLATE=utf8_unicode_ci;

call populate('test','movies_catalog',50000,'N');

# Problem
begin;
update movies_catalog set catalog_id=catalog_id+1;
call populate('test','movies_catalog',10000,'N');
update movies_catalog set catalog_id=catalog_id+10;
update movies_catalog set catalog_id=catalog_id+10;
update movies_catalog set catalog_id=catalog_id+10;
update movies_catalog set catalog_id=catalog_id+10 limit 1;
update movies_catalog set catalog_id=catalog_id+10 limit 10;
update movies_catalog set catalog_id=catalog_id+10 limit 100;
commit;

Suggested fix:
Not sure if there are plans to add patches on bug report 72457 or another fix for version 8.0.

Replication should not break for a committed transaction on Source.
[21 Jul 2022 13:27] MySQL Verification Team
Hi,

I verified the behavior but note it is much easier to monitor your servers using MySQL Enterprise manager (or any other server monitoring tools that any decent sysadmin should setup to monitor disk usage and warn if free space drops below a point) than to fix this behavior.

all best