MySQL Bugs: #43911: Innodb crash for no obvious reason

Bug #43911	Innodb crash for no obvious reason
Submitted:	27 Mar 2009 13:52	Modified:	25 Jan 2010 11:36
Reporter:	Kiriakos Tsourapas	Email Updates:
Status:	No Feedback	Impact on me:	None
Category:	MySQL Server: InnoDB storage engine	Severity:	S2 (Serious)
Version:	5.0.27-standard	OS:	Linux
Assigned to:		CPU Architecture:	Any

Description:
We upgraded from SUSE Enterprise Server 10 to 10SP2. Nevertheless, we have 3 identical installations and the problem occurred only in one of the three, so I guess the OS upgrade is not the problem, but I am mentioning it...

Innodb crashed after about 10 hours of being up.
It recovered and continued, but we lost our application for about 2 minutes.

I am sending any useful info, so that you can check it.

The disk space is free by 90G.

Will attach the error log after the bug is created...

How to repeat:
No way to repeat it, since we have no idea why it happened.

The system was under regular load (about 20 TPS).
Peak time he have about 40 TPS.

THE ERROR LOG
=============
090325 01:59:37 mysqld started
090325 1:59:38 InnoDB: Started; log sequence number 1837 635594034
090325 1:59:38 [Note] /usr/local/mysql/bin/mysqld: ready for connections.
Version: '5.0.27-standard-log' socket: '/tmp/mysql.sock' port: 3306 MySQL Community Edition - Standard (GPL)
090325 11:32:35 InnoDB: Error: Write to file ./ibdata1 failed at offset 2 1180696576.
InnoDB: 999424 bytes should have been written, only 462848 were written.
InnoDB: Operating system error number 0.
InnoDB: Check that your OS and file system support files of this size.
InnoDB: Check also that the disk is not full or a disk quota exceeded.
InnoDB: Error number 0 means 'Success'.
InnoDB: Some operating system error numbers are described at
InnoDB: http://dev.mysql.com/doc/refman/5.0/en/operating-system-error-codes.html
090325 11:32:35InnoDB: Assertion failure in thread 1107310912 in file os0file.c line 3991
InnoDB: Failing assertion: ret
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: http://dev.mysql.com/doc/refman/5.0/en/forcing-recovery.html
InnoDB: about forcing recovery.
mysqld got signal 11;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help diagnose
the problem, but since we have already crashed, something is definitely wrong
and this may fail.

Full Error Log

Attachment: error.log (application/octet-stream, text), 10.38 KiB.

my.cnf

Attachment: my.cnf (application/octet-stream, text), 5.27 KiB.

System Variables

Attachment: variables.txt (text/plain), 17.52 KiB.

Table Statuses of the most used database

Attachment: table_status.txt (text/plain), 4.18 KiB.

Thank you for the bug report. Your server version is quite older and a lot of bug fixes were applied since that ancient version. Could you please try to upgrade after reading the Manual's release note section. Thanks in advance.

We know our version is old and that we need to upgrade, but we have the same installation to 3 different sites, 8 different servers. It is quite hard to upgrade right now. It is scheduled, but not for now.

The reason I posted the bug is whether from the logs you can see something we don't and maybe re-configure MySQL so that it does not appear again.

Furthermore, upgrading 8 servers, only hoping to solve an issue is not a real good idea. Can you at least verify that the issue is solved in a later release ?

Thank you for your support.

Thank you for the feedback. Sorry I wasn`t able to find a similar case reported and had you checked for hard disk space and integrity?. Thanks in advance.

There are 90Gb free, so it's definitely not the disk space.
Our technicians did not see any problems with the hardware either.

So, I am gu

Sorry, submitted by mistake...

There are 90Gb free, so it's definitely not the disk space.
Our technicians did not see any problems with the hardware either.

So, I am guessing that there is no way to figure out what caused it and to avoid future re-appearance ?

Thank you for the feedback.

Yes, information is not enough to repeat the problem. If this is still repeatable in your environment please try to get more information about the problem: use debug version, turn generation of core files to on (you should start mysqld with option --core-file and adjust OS settings accordingly) and try to find a query which causes failure.

No feedback was provided for this bug for over a month, so it is
being suspended automatically. If you are able to provide the
information that was originally requested, please do so and change
the status of the bug back to "Open".