Bug #71366 MySQL Server Crashes
Submitted: 13 Jan 2014 23:23 Modified: 27 Jan 2014 13:37
Reporter: Frederic Steinfels Email Updates:
Status: Not a Bug Impact on me:
None 
Category:MySQL Server: InnoDB storage engine Severity:S3 (Non-critical)
Version:5.6.15 OS:Linux (Fedora 20)
Assigned to: CPU Architecture:Any
Tags: 1048576, crash, disk, full, innodb, MySQL, OFFSET

[13 Jan 2014 23:23] Frederic Steinfels
Description:
as of late, I am getting more and more crashes:

[root@www log]# tail -n10000 -f mysqld.log | grep offset
2014-01-07 20:12:53 7f28f5d8f700 InnoDB: Error: Write to file ./ibdata1 failed at offset 1048576.
2014-01-08 19:17:43 7f302ffff700 InnoDB: Error: Write to file ./ibdata1 failed at offset 1048576.
2014-01-08 20:18:09 7f1fd37fe700 InnoDB: Error: Write to file ./ibdata1 failed at offset 1048576.
2014-01-12 01:34:43 7f96d3e8b700 InnoDB: Error: Write to file ./ibdata1 failed at offset 1048576.
2014-01-12 12:37:47 7fa9effff700 InnoDB: Error: Write to file ./ib_logfile0 failed at offset 8882176.
2014-01-12 18:39:46 7f6d96ffd700 InnoDB: Error: Write to file ./ibdata1 failed at offset 1048576.
2014-01-12 23:40:57 7f85c7fff700 InnoDB: Error: Write to file ./ibdata1 failed at offset 1048576.

The stacktrace is always identical:

2014-01-12 23:40:57 7f85c7fff700 InnoDB: Error: Write to file ./ibdata1 failed at offset 1048576.
InnoDB: 16384 bytes should have been written, only -1 were written.
InnoDB: Operating system error number 28.
InnoDB: Check that your OS and file system support files of this size.
InnoDB: Check also that the disk is not full or a disk quota exceeded.
InnoDB: Error number 28 means 'No space left on device'.
InnoDB: Some operating system error numbers are described at
InnoDB: http://dev.mysql.com/doc/refman/5.6/en/operating-system-error-codes.html
2014-01-12 23:40:57 7f85c7fff700  InnoDB: Assertion failure in thread 140212562818816 in file fil0fil.cc line 5523
InnoDB: Failing assertion: ret
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: http://dev.mysql.com/doc/refman/5.6/en/forcing-innodb-recovery.html
InnoDB: about forcing recovery.
22:40:57 UTC - mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.

key_buffer_size=8388608
read_buffer_size=131072
max_used_connections=25
max_threads=200
thread_count=16
connection_count=16
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 87729 K  bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0x0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0 thread_stack 0x80000
/usr/sbin/mysqld(my_print_stacktrace+0x2c)[0x8ca23c]
/usr/sbin/mysqld(handle_fatal_signal+0x491)[0x669f11]
/lib64/libpthread.so.0[0x3b2d00ef90]
/lib64/libc.so.6(gsignal+0x39)[0x3b2cc359e9]
/lib64/libc.so.6(abort+0x148)[0x3b2cc370f8]
/usr/sbin/mysqld[0xa46675]
/usr/sbin/mysqld[0xa01384]
/usr/sbin/mysqld[0xa07844]
/usr/sbin/mysqld[0xa08b54]
/lib64/libpthread.so.0[0x3b2d007c53]
/lib64/libc.so.6(clone+0x6d)[0x3b2ccf5dbd]
The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you find out what is causing the crash.
140112 23:40:57 mysqld_safe Number of processes running now: 0
140112 23:40:57 mysqld_safe mysqld restarted
2014-01-12 23:40:58 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestamp server option (see documentation for more details).
2014-01-12 23:40:58 6402 [Note] Plugin 'FEDERATED' is disabled.
2014-01-12 23:40:58 7f04e6578740 InnoDB: Warning: Using innodb_locks_unsafe_for_binlog is DEPRECATED. This option may be removed in future releases. Please use READ COMMITTED transaction isolation level instead, see http://dev.mysql.com/doc/refman/5.6/en/set-transaction.html.
2014-01-12 23:40:58 6402 [Note] InnoDB: The InnoDB memory heap is disabled
2014-01-12 23:40:58 6402 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
2014-01-12 23:40:58 6402 [Note] InnoDB: Compressed tables use zlib 1.2.3
2014-01-12 23:40:58 6402 [Note] InnoDB: Using Linux native AIO
2014-01-12 23:40:58 6402 [Note] InnoDB: Using CPU crc32 instructions
2014-01-12 23:40:58 6402 [Note] InnoDB: Initializing buffer pool, size = 20.0G
2014-01-12 23:40:58 6402 [Note] InnoDB: Completed initialization of buffer pool
2014-01-12 23:40:58 6402 [Note] InnoDB: Highest supported file format is Barracuda.
2014-01-12 23:40:58 6402 [Note] InnoDB: Log scan progressed past the checkpoint lsn 1274555890716
2014-01-12 23:40:58 6402 [Note] InnoDB: Database was not shutdown normally!
2014-01-12 23:40:58 6402 [Note] InnoDB: Starting crash recovery.
2014-01-12 23:40:58 6402 [Note] InnoDB: Reading tablespace information from the .ibd files...
2014-01-12 23:40:58 6402 [Note] InnoDB: Restoring possible half-written data pages
2014-01-12 23:40:58 6402 [Note] InnoDB: from the doublewrite buffer...
InnoDB: Doing recovery: scanned up to log sequence number 1274561133568
InnoDB: Doing recovery: scanned up to log sequence number 1274564034382
2014-01-12 23:40:59 6402 [Note] InnoDB: Starting an apply batch of log records to the database...
InnoDB: Progress in percent: 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99
InnoDB: Apply batch completed
2014-01-12 23:41:00 6402 [Note] InnoDB: 128 rollback segment(s) are active.
2014-01-12 23:41:01 6402 [Note] InnoDB: Waiting for purge to start
2014-01-12 23:41:01 6402 [Note] InnoDB: 5.6.15 started; log sequence number 1274564034382
2014-01-12 23:41:01 6402 [Note] Server hostname (bind-address): '*'; port: 3306
2014-01-12 23:41:01 6402 [Note] IPv6 is available.
2014-01-12 23:41:01 6402 [Note]   - '::' resolves to '::';
2014-01-12 23:41:01 6402 [Note] Server socket created on IP: '::'.
2014-01-12 23:41:01 6402 [Note] Event Scheduler: Loaded 0 events
2014-01-12 23:41:01 6402 [Note] /usr/sbin/mysqld: ready for connections.
Version: '5.6.15-log'  socket: '/var/lib/mysql/mysql.sock'  port: 3306  MySQL Community Server (GPL)
2014-01-12 23:41:01 6402 [Warning] Hostname 'www.highdefinition.ch' does not resolve to '192.168.99.15'.
2014-01-12 23:41:01 6402 [Note] Hostname 'www.highdefinition.ch' has the following IP addresses:
2014-01-12 23:41:01 6402 [Note]  - 144.76.167.66

The file System is far from being full:

[root@www log]# df
Filesystem      1K-blocks       Used Available Use% Mounted on
/dev/sda2       464760832  235697280 226281480  52% /

Although it is a btrfs partition

How to repeat:
sit back and wait... sorry

Suggested fix:
don't crash
[13 Jan 2014 23:26] Frederic Steinfels
updated
[23 Jan 2014 0:20] MySQL Verification Team
if innodb cannot write to disk, in this case OS told it out of space, it will crash.  Sorry, this isn't a bug in the server as far as I could see.  Do you have any other monitoring, or looked in system logs /var/log/message* for further info?
[27 Jan 2014 13:37] Frederic Steinfels
I have placed a bug report on kernel.org
https://bugzilla.kernel.org/show_bug.cgi?id=69321
As there was no reply and the same error occured when using the 'chown' command, I have revereted back to ext4.
Thanks for your time.