Description:
Hi,
I try to understand [1], and this lead me to this bug. So this bug is a joint effort by Mydbops, Waseem Akram and me.
[1]: https://mydbops.wordpress.com/2019/02/22/impact-of-tmpdir-change-in-mysql-replication
When an ALTER TABLE fails because not being able to write to tmdir, there is a scary "Slave: Out of memory" message in the logs (see below and How to repeat). But I am not able to reproduce the corruption situation of [1] (yet) and restarting MySQL with a fixed tmpdir allows to resume replication.
2019-02-23 09:01:31 8770 [Warning] Slave: Out of memory; check if mysqld or some other process uses all available memory; if not, you may have to use 'ulimit' to allow mysqld to use more memory or you can add more swap space Error_code: 1041
Thanks in advance for looking into that,
JFG
How to repeat:
$ dbdeployer deploy --repl-crash-safe replication mysql_5.6.42
[...]
$ cd $sandbox_dir
$ ./m <<< "create database test_jfg"
$ ./m test_jfg <<< "
create table t1(id int primary key, t varchar(20));
insert into t1 value(1, 'hello world')"
$ chmod a-rwx node1/tmp/
$ ./m test_jfg <<< "alter table t1 add index(t)"
$ tail -n 5 node1/data/msandbox.err
2019-02-23 09:01:31 8770 [ERROR] InnoDB: Cannot create temporary merge file
2019-02-23 09:01:31 8770 [ERROR] Slave SQL: Error 'Can't create/write to file '/home/jgagne/sandboxes/rsandbox_mysql_5_6_42/node1/tmp/ibGkl1mD' (Errcode: 13 - Permission denied)' on query. Default database: 'test_jfg'. Query: 'alter table t1 add index(t)', Error_code: 1
2019-02-23 09:01:31 8770 [Warning] Slave: Can't create/write to file '/home/jgagne/sandboxes/rsandbox_mysql_5_6_42/node1/tmp/ibGkl1mD' (Errcode: 13 - Permission denied) Error_code: 1
2019-02-23 09:01:31 8770 [Warning] Slave: Out of memory; check if mysqld or some other process uses all available memory; if not, you may have to use 'ulimit' to allow mysqld to use more memory or you can add more swap space Error_code: 1041
2019-02-23 09:01:31 8770 [ERROR] Error running query, slave SQL thread aborted. Fix the problem, and restart the slave SQL thread with "SLAVE START". We stopped at log 'mysql-bin.000001' position 3174.
-- "Slave: Out of memory": really !
