MySQL Bugs: #94289: MySQL Server Crashing

Bug #94289	MySQL Server Crashing
Submitted:	12 Feb 2019 10:07	Modified:	14 Feb 2019 11:23
Reporter:	Leonardo Amorim	Email Updates:
Status:	Not a Bug	Impact on me:	None
Category:	MySQL Server: Errors	Severity:	S2 (Serious)
Version:	5.7.21	OS:	SUSE (OpenSuse 13.2)
Assigned to:		CPU Architecture:	Any

Description:
I'm having a problem in a production server. My server is rebooting in aleatory times. I checked my disk with df -h and it's working with only 56% of space used. But sometimes the server crash and try to restart.

The following error occurs:

2019-02-12T02:26:15.412830Z 156530 [Warning] InnoDB: Retry attempts for writing partial data failed.
2019-02-12T02:26:17.432993Z 156530 [ERROR] InnoDB: Write to file ./ib_logfile0failed at offset 1791099392, 2048 bytes should have been written, only 0 were written. Operating system error number 28. Check that your OS and file system support files of this size. Check also that the disk is not full or a disk quota exceeded.
2019-02-12T02:26:17.708155Z 156530 [ERROR] InnoDB: Error number 28 means 'No space left on device'
2019-02-12T02:26:18.417133Z 156530 [Note] InnoDB: Some operating system error numbers are described at http://dev.mysql.com/doc/refman/5.7/en/operating-system-error-codes.html
2019-02-12 00:26:19 0x7fc4b0b29700  InnoDB: Assertion failure in thread 140482754811648 in file fil0fil.cc line 5799
InnoDB: Failing assertion: req_type.is_dblwr_recover() || err == DB_SUCCESS
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: http://dev.mysql.com/doc/refman/5.7/en/forcing-innodb-recovery.html
InnoDB: about forcing recovery.
02:26:20 UTC - mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
Attempting to collect some information that could help diagnose the problem.
As this is a crash and something is definitely wrong, the information
collection process might fail.

key_buffer_size=2621440000
read_buffer_size=262144
max_used_connections=121
max_threads=1200
thread_count=65
connection_count=64
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 52035231 K  bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0x7fc4500c36f0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 7fc4b0b28e80 thread_stack 0x40000
/usr/sbin/mysqld(my_print_stacktrace+0x2c)[0xf16e3c]
/usr/sbin/mysqld(handle_fatal_signal+0x451)[0x7fa791]
/lib64/libpthread.so.0(+0xf890)[0x7fcea4809890]
/lib64/libc.so.6(gsignal+0x37)[0x7fcea300a187]
/lib64/libc.so.6(abort+0x118)[0x7fcea300b538]
/usr/sbin/mysqld[0x7cb390]
/usr/sbin/mysqld(_Z6fil_ioRK9IORequestbRK9page_id_tRK11page_size_tmmPvS8_+0x772)[0x125e762]
/usr/sbin/mysqld(_Z15log_write_up_tomb+0x734)[0x1063b04]
/usr/sbin/mysqld(_Z24log_buffer_flush_to_diskb+0x108)[0x1064648]
/usr/sbin/mysqld[0xffe3ff]
/usr/sbin/mysqld[0x8406ef]
/usr/sbin/mysqld(_Z24plugin_foreach_with_maskP3THDPPFcS0_P13st_plugin_intPvEijS3_+0x1fe)[0xd0ebfe]
/usr/sbin/mysqld(_Z24plugin_foreach_with_maskP3THDPFcS0_P13st_plugin_intPvEijS3_+0x1d)[0xd0ed7d]
/usr/sbin/mysqld(_Z13ha_flush_logsP10handlertonb+0x5a)[0x846b7a]
/usr/sbin/mysqld(_ZN13MYSQL_BIN_LOG25process_flush_stage_queueEPyPbPP3THD+0x3b)[0xeb327b]
/usr/sbin/mysqld(_ZN13MYSQL_BIN_LOG14ordered_commitEP3THDbb+0x10f)[0xeb344f]
/usr/sbin/mysqld(_ZN13MYSQL_BIN_LOG6commitEP3THDb+0x994)[0xeb66d4]
/usr/sbin/mysqld(_Z15ha_commit_transP3THDbb+0x131)[0x846171]
/usr/sbin/mysqld(_Z12trans_commitP3THD+0x39)[0xd91ec9]
/usr/sbin/mysqld(_Z21mysql_execute_commandP3THDb+0x625)[0xce9965]
/usr/sbin/mysqld(_Z11mysql_parseP3THDP12Parser_state+0x3bd)[0xcefc3d]
/usr/sbin/mysqld(_Z16dispatch_commandP3THDPK8COM_DATA19enum_server_command+0xaba)[0xcf07ca]
/usr/sbin/mysqld(_Z10do_commandP3THD+0x177)[0xcf21f7]
/usr/sbin/mysqld(handle_connection+0x270)[0xdae930]
/usr/sbin/mysqld(pfs_spawn_thread+0x1b4)[0xf91674]
/lib64/libpthread.so.0(+0x80a4)[0x7fcea48020a4]
/lib64/libc.so.6(clone+0x6d)[0x7fcea30ba7fd]

Trying to get some variables.
Some pointers may be invalid and cause the dump to abort.
Query (7fc45004f770): commit
Connection ID (thread ID): 156530
Status: NOT_KILLED

The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you find out what is causing the crash.
2019-02-12T02:26:34.937708Z 0 [Warning] Changed limits: max_open_files: 5000 (requested 6000)

How to repeat:
This occurs in indefinite periods. This morning the server rebooted at 2AM. At this time there is no access to the databases.

Changing the category

Hi,

InnoDB storage engine has got the errno of 28. That only means that you have ran out of the disk space.

InnoDB must crash the server, because, otherwise, data on disk would be corrupted.

You might see lot's of space free on the disk, when you get this error, but you could still run out of disk space, because of one or several of those factors:

* You have  partitioned your disk and you are using only one (or some) of the partitions
* MySQL server is hosted by third party and you have ran out of the allowed space
* You have ran out of the disk quota for the user ID under which MySQL server is running ..
* MySQL server had created lots of temporary tables, which used the free disk space, but were deleted during the crash.

Simply, this is not a bug.

Hi Sinisa.

Thank's for your time. Some of these points we don't use in our environment.

* You have  partitioned your disk and you are using only one (or some) of the partitions
A: Our disk has only one partition (/dev/sda2 with: 303G, available: 141G)

* MySQL server is hosted by third party and you have ran out of the allowed space
A: We use a virtualized server MySQL 5.7.21 Community Server (GPL), the file system is btrfs

* You have ran out of the disk quota for the user ID under which MySQL server is running ..
A: We don't use quota do read/write in SO

* MySQL server had created lots of temporary tables, which used the free disk space, but were deleted during the crash.
A: SELECT * FROM INFORMATION_SCHEMA.INNODB_TEMP_TABLE_INFO. It does not return any result.

Hi,

First of all, all virtualised environments set some limits, including a disk limit.

Second, getting a report on temporary tables AFTER those have been removed provides nothing useful.

Third, and most important, errno 28 is the one that our server gets from the operating system. We have to react to I/O errors that OS reports. Hence, that is your problem.

Definitely, it's a disk problem. The solution was to create a new server CentOS and overwrite the production server. In sequence after mysqldump finished, the server crashed and didn't start anymore.

I'm closing this topic.

Thank you for your confirmation.