MySQL Bugs: #88262: ERROR 1598 (HY000): Binary logging not possible + abort

Bug #88262	ERROR 1598 (HY000): Binary logging not possible + abort
Submitted:	28 Oct 2017 0:17	Modified:	19 Dec 2017 12:08
Reporter:	Roel Van de Paar	Email Updates:
Status:	Closed	Impact on me:	None
Category:	MySQL Server: Replication	Severity:	S1 (Critical)
Version:	8.0.3 RC	OS:	Any
Assigned to:		CPU Architecture:	Any

Description:
8.0.3>FLUSH BINARY LOGS;
ERROR 1598 (HY000): Binary logging not possible. Message: Either disk is full or file system is read only while rotating the binlog. Aborting the server.

Neither is true.

2017-10-27T23:45:42.971340Z 0 [Note] /sda/MS201017-mysql-8.0.3-rc-linux-x86_64-debug/bin/mysqld: ready for connections. Version: '8.0.3-rc-debug-log'  socket: '/sda/MS201017-mysql-8.0.3-rc-linux-x86_64-debug/socket.sock'  port: 16654  MySQL Community Server (GPL)
2017-10-27T23:46:32.175582Z 7 [Warning] Next log extension: 2147483647. Remaining log filename extensions: 0. Please consider archiving some logs.
2017-10-27T23:46:32.188457Z 7 [Warning] Next log extension: 2147483647. Remaining log filename extensions: 0. Please consider archiving some logs.
2017-10-27T23:46:32.209581Z 7 [ERROR] Log filename extension number exhausted: 2147483647. Please fix this by archiving old logs and updating the index files.
2017-10-27T23:46:32.209654Z 7 [ERROR] Can't generate a unique log-filename localhost-bin.(1-999)

2017-10-27T23:46:32.209692Z 7 [ERROR] /sda/MS201017-mysql-8.0.3-rc-linux-x86_64-debug/bin/mysqld: Binary logging not possible. Message: Either disk is full or file system is read only while rotating the binlog. Aborting the server.
23:46:32 UTC - mysqld got signal 6 ;

Core was generated by `/sda/MS201017-mysql-8.0.3-rc-linux-x86_64-debug/bin/mysqld --no-defaults --core'.
Program terminated with signal 6, Aborted.
#0  0x00007f9ec96849b1 in __pthread_kill (threadid=<optimized out>, signo=6) at ../nptl/sysdeps/unix/sysv/linux/pthread_kill.c:61
61	  val = INTERNAL_SYSCALL (tgkill, err, 3, THREAD_GETMEM (THREAD_SELF, pid),
(gdb) bt
#0  0x00007f9ec96849b1 in __pthread_kill (threadid=<optimized out>, signo=6) at ../nptl/sysdeps/unix/sysv/linux/pthread_kill.c:61
#1  0x000000000369e428 in my_write_core (sig=6) at /git/mysql-server_dbg/mysys/stacktrace.cc:291
#2  0x000000000286b8f6 in handle_fatal_signal (sig=6) at /git/mysql-server_dbg/sql/signal_handler.cc:234
#3  <signal handler called>
#4  0x00007f9ec799c1f7 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
#5  0x00007f9ec799d8e8 in __GI_abort () at abort.c:90
#6  0x00000000035d7668 in exec_binlog_error_action_abort (
    err_string=0x4781db0 "Either disk is full or file system is read only while rotating the binlog. Aborting the server.")
    at /git/mysql-server_dbg/sql/binlog.cc:2205
#7  0x00000000035e5854 in MYSQL_BIN_LOG::new_file_impl (this=0x56e3f80 <mysql_bin_log>, need_lock_log=false, extra_description_event=
    0x0) at /git/mysql-server_dbg/sql/binlog.cc:7220
#8  0x00000000035e5019 in MYSQL_BIN_LOG::new_file_without_locking (this=0x56e3f80 <mysql_bin_log>, extra_description_event=0x0)
    at /git/mysql-server_dbg/sql/binlog.cc:7016
#9  0x00000000035e7187 in MYSQL_BIN_LOG::rotate (this=0x56e3f80 <mysql_bin_log>, force_rotate=true, check_purge=0x7f9ec9c290eb)
    at /git/mysql-server_dbg/sql/binlog.cc:7704
#10 0x00000000035e7355 in MYSQL_BIN_LOG::rotate_and_purge (this=0x56e3f80 <mysql_bin_log>, thd=0x7f9e80c19000, force_rotate=true)
    at /git/mysql-server_dbg/sql/binlog.cc:7764
#11 0x00000000024ffd38 in reload_acl_and_cache (thd=0x7f9e80c19000, options=1024, tables=0x0, write_to_binlog=0x7f9ec9c29e7c)
    at /git/mysql-server_dbg/sql/sql_reload.cc:199
#12 0x00000000024bcb84 in mysql_execute_command (thd=0x7f9e80c19000, first_level=true) at /git/mysql-server_dbg/sql/sql_parse.cc:4070
#13 0x00000000024c042b in mysql_parse (thd=0x7f9e80c19000, parser_state=0x7f9ec9c2a2f0) at /git/mysql-server_dbg/sql/sql_parse.cc:5414
#14 0x00000000024b68e7 in dispatch_command (thd=0x7f9e80c19000, com_data=0x7f9ec9c2ab80, command=COM_QUERY)
    at /git/mysql-server_dbg/sql/sql_parse.cc:1708
#15 0x00000000024b53f5 in do_command (thd=0x7f9e80c19000) at /git/mysql-server_dbg/sql/sql_parse.cc:1291
#16 0x000000000285b2d6 in handle_connection (arg=0x7f9eb5aeb3c0)
    at /git/mysql-server_dbg/sql/conn_handler/connection_handler_per_thread.cc:328
#17 0x00000000036ec54d in pfs_spawn_thread (arg=0x7f9eb5b85520) at /git/mysql-server_dbg/storage/perfschema/pfs.cc:2985
#18 0x00007f9ec967fe25 in start_thread (arg=0x7f9ec9c2b700) at pthread_create.c:308
#19 0x00007f9ec7a5f34d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113

How to repeat:
RESET MASTER TO 2147483647;
FLUSH BINARY LOGS;
SELECT 1;  # Shows server is gone

Am I missing something here?

mysql> RESET MASTER TO 2147483647;
ERROR 1064 (42000): You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'TO 2147483647' at line 1

If you are manually renaming the binlog files so mysqld thinks you've run out of numbers,  then the error before crash is expected and self-explanatory.  
Please confirm.

No manual renaming, only cli input. My output differs too;

Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 7
Server version: 8.0.3-rc-debug-log MySQL Community Server (GPL)

Copyright (c) 2000, 2017, Oracle and/or its affiliates. All rights reserved.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

8.0.3>RESET MASTER TO 2147483647;
Query OK, 0 rows affected (0.06 sec)

8.0.3>FLUSH BINARY LOGS;
ERROR 1598 (HY000): Binary logging not possible. Message: Either disk is full or file system is read only while rotating the binlog. Aborting the server.
8.0.3>SELECT 1;  # Shows server is gone
ERROR 2013 (HY000): Lost connection to MySQL server during query

mysqld started with no special options;
mysqld --no-defaults --core-file --basedir=... --tmpdir=... --datadir=...   --socket=... --port=... --log-error=... 2>&1 &

In case it matters;
-bash-4.2$ cat /etc/centos-release
CentOS Linux release 7.4.1708 (Core) 
-bash-4.2$ free -g
              total        used        free      shared  buff/cache   available
Mem:             31           9           5          15          16           5
Swap:            31           5          25

Thanks!
[NOTE]mysqld-debug.exe: Version: '8.0.4-rc-debug-log' (Built on 2017/10/14)

[Warning] [004438] Next log extension: 2147483647. Remaining log filename extensions: 0. Please consider archiving some logs.
[Warning] [004438] Next log extension: 2147483647. Remaining log filename extensions: 0. Please consider archiving some logs.
[ERROR] [004436] Log filename extension number exhausted: 2147483647. Please fix this by archiving old logs and updating the index files.
[ERROR] [001098] Can't generate a unique log-filename i7-bin.(1-999)

mysqld-debug.exe!my_sigabrt_handler()[my_thr_init.cc:418]
  ucrtbased.dll!raise()
  ucrtbased.dll!abort()
mysqld-debug.exe!exec_binlog_error_action_abort()[binlog.cc:2206]
mysqld-debug.exe!MYSQL_BIN_LOG::new_file_impl()[binlog.cc:7222]
mysqld-debug.exe!MYSQL_BIN_LOG::new_file_without_locking()[binlog.cc:7017]
mysqld-debug.exe!MYSQL_BIN_LOG::rotate()[binlog.cc:7704]
mysqld-debug.exe!MYSQL_BIN_LOG::rotate_and_purge()[binlog.cc:7770]
mysqld-debug.exe!reload_acl_and_cache()[sql_reload.cc:199]
mysqld-debug.exe!mysql_execute_command()[sql_parse.cc:4061]
mysqld-debug.exe!mysql_parse()[sql_parse.cc:5407]
mysqld-debug.exe!dispatch_command()[sql_parse.cc:1708]
mysqld-debug.exe!do_command()[sql_parse.cc:1292]
mysqld-debug.exe!handle_connection()[connection_handler_per_thread.cc:328]
mysqld-debug.exe!pfs_spawn_thread()[pfs.cc:2989]
mysqld-debug.exe!win_thread_start()[my_thread.cc:42]

Posted by developer:
 
Changelog entry added for MySQl 8.0.5:

The error message issued for a server stop while rotating the binary log did not state the correct cause for the issue. The error message has been enhanced to include the actual error causing the server stop as part of the message. A new error ER_OOM_SAVE_GTIDS has also been added for the situation where an out-of-memory error occured while saving the set of GTIDs from the last binary log into the mysql.gtid_executed table.