MySQL Bugs: #43911: Innodb crash for no obvious region

Bug #43911	Innodb crash for no obvious region
Submitted:	27 Mar 14:52	Modified:	20 Aug 19:14
Reporter:	Kyriakos Tsourapas
Status:	Open
Category:	Server: InnoDB	Severity:	S2 (Serious)
Version:	5.0.27-standard	OS:	Linux
Assigned to:		Target Version:

Description:
We upgraded from SUSE Enterprise Server 10 to 10SP2. Nevertheless, we have 3 identical
installations and the problem occurred only in one of the three, so I guess the OS
upgrade is not the problem, but I am mentioning it...

Innodb crashed after about 10 hours of being up.
It recovered and continued, but we lost our application for about 2 minutes.

I am sending any useful info, so that you can check it.

The disk space is free by 90G.

Will attach the error log after the bug is created...

How to repeat:
No way to repeat it, since we have no idea why it happened.

The system was under regular load (about 20 TPS).
Peak time he have about 40 TPS.

THE ERROR LOG
=============
090325 01:59:37  mysqld started
090325  1:59:38  InnoDB: Started; log sequence number 1837 635594034
090325  1:59:38 [Note] /usr/local/mysql/bin/mysqld: ready for connections.
Version: '5.0.27-standard-log'  socket: '/tmp/mysql.sock'  port: 3306  MySQL Community
Edition - Standard (GPL)
090325 11:32:35  InnoDB: Error: Write to file ./ibdata1 failed at offset 2 1180696576.
InnoDB: 999424 bytes should have been written, only 462848 were written.
InnoDB: Operating system error number 0.
InnoDB: Check that your OS and file system support files of this size.
InnoDB: Check also that the disk is not full or a disk quota exceeded.
InnoDB: Error number 0 means 'Success'.
InnoDB: Some operating system error numbers are described at
InnoDB: http://dev.mysql.com/doc/refman/5.0/en/operating-system-error-codes.html
090325 11:32:35InnoDB: Assertion failure in thread 1107310912 in file os0file.c line
3991
InnoDB: Failing assertion: ret
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: http://dev.mysql.com/doc/refman/5.0/en/forcing-recovery.html
InnoDB: about forcing recovery.
mysqld got signal 11;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help diagnose
the problem, but since we have already crashed, something is definitely wrong
and this may fail.

Full Error Log

Attachment: error.log (application/octet-stream, text), 10.38 KiB.

my.cnf

Attachment: my.cnf (application/octet-stream, text), 5.27 KiB.

System Variables

Attachment: variables.txt (text/plain), 17.52 KiB.

Table Statuses of the most used database

Attachment: table_status.txt (text/plain), 4.18 KiB.

Thank you for the bug report. Your server version is quite older and a lot of bug fixes
were applied since that ancient version. Could you please try to upgrade after reading
the Manual's release note section. Thanks in advance.

We know our version is old and that we need to upgrade, but we have the same installation
to 3 different sites, 8 different servers. It is quite hard to upgrade right now. It is
scheduled, but not for now.

The reason I posted the bug is whether from the logs you can see something we don't and
maybe re-configure MySQL so that it does not appear again.

Furthermore, upgrading 8 servers, only hoping to solve an issue is not a real good idea.
Can you at least verify that the issue is solved in a later release ?

Thank you for your support.

Thank you for the feedback. Sorry I wasn`t able to find a similar case reported and had
you checked for hard disk space and integrity?. Thanks in advance.

There are 90Gb free, so it's definitely not the disk space.
Our technicians did not see any problems with the hardware either.

So, I am gu

Sorry, submitted by mistake...

There are 90Gb free, so it's definitely not the disk space.
Our technicians did not see any problems with the hardware either.

So, I am guessing that there is no way to figure out what caused it and to avoid future
re-appearance ?