MySQL Bugs: #31046: mysqld kernel panic

Bug #31046	mysqld kernel panic
Submitted:	16 Sep 2007 13:26	Modified:	14 Jan 2008 5:42
Reporter:	Lukasz Kawczynski	Email Updates:
Status:	No Feedback	Impact on me:	None
Category:	MySQL Server: General	Severity:	S2 (Serious)
Version:	5.0.45	OS:	Linux (Red Hat Enterprise Linux ES release 4 (Nahant Update 5))
Assigned to:		CPU Architecture:	Any
Tags:	kernel, panic

Description:
One of my servers keeps crashing about 4-5 times a day.

Logs contain information about 34 kernel panics involving mysqld.

Sep 16 00:55:27 ev1s-xxx kernel: Unable to handle kernel paging request at virtual address 00fb3000
Sep 16 00:55:27 ev1s-xxx kernel:  printing eip:
Sep 16 00:55:27 ev1s-xxx kernel: f88bb6f1
Sep 16 00:55:27 ev1s-xxx kernel: *pde = 00000000
Sep 16 00:55:27 ev1s-xxx kernel: Oops: 0003 [#1]
Sep 16 00:55:27 ev1s-xxx kernel: SMP 
Sep 16 00:55:27 ev1s-xxx kernel: Modules linked in: md5 ipv6 autofs4 sunrpc ip_conntrack_ftp ipt_state ip_conntrack iptable_filter ip_tables dm_mirror dm_mod button battery ac ohci_hcd e1000 floppy ext3 jbd mptscsih mptsas mptspi mptfc mptscsi mptbase sd_mod scsi_mod
Sep 16 00:55:27 ev1s-xxx kernel: CPU:    1
Sep 16 00:55:27 ev1s-xxx kernel: EIP:    0060:[<f88bb6f1>]    Not tainted VLI
Sep 16 00:55:27 ev1s-xxx kernel: EFLAGS: 00010246   (2.6.9-55.0.6.ELsmp) 
Sep 16 00:55:27 ev1s-xxx kernel: EIP is at 0xf88bb6f1
Sep 16 00:55:27 ev1s-xxx kernel: eax: 00000000   ebx: 00048000   ecx: ffffffea   edx: 00000000
Sep 16 00:55:27 ev1s-xxx kernel: esi: 00fb3000   edi: 007a6ff4   ebp: e89f8000   esp: e89f8e90
Sep 16 00:55:27 ev1s-xxx kernel: ds: 007b   es: 007b   ss: 0068
Sep 16 00:55:27 ev1s-xxx kernel: Process mysqld (pid: 17134, threadinfo=e89f8000 task=f138d630)
Sep 16 00:55:27 ev1s-xxx kernel: Stack: 00000000 00047000 00000000 f88bb7e8 00047000 00000002 00000000 007a6ff4 
Sep 16 00:55:27 ev1s-xxx kernel:        00000000 0000001f 00000000 f88bb9d0 00047000 e89f8f6c 00000801 f88bb9a7 
Sep 16 00:55:27 ev1s-xxx kernel:        f88b558a 00000000 00000000 f88c54ac 00047000 0000001f 00000000 00000001 
Sep 16 00:55:27 ev1s-xxx kernel: Call Trace:
Sep 16 00:55:27 ev1s-xxx kernel:  [<c0163936>] cp_new_stat64+0x125/0x13a
Sep 16 00:55:27 ev1s-xxx kernel:  [<c016a902>] set_close_on_exec+0x1e/0x38
Sep 16 00:55:27 ev1s-xxx kernel:  [<c016adfb>] do_fcntl+0xec/0x15f
Sep 16 00:55:27 ev1s-xxx kernel:  [<c016af37>] sys_fcntl64+0x6f/0x80
Sep 16 00:55:28 ev1s-xxx kernel: Code: 74 32 50 e8 00 00 00 00 58 2d 44 61 00 00 53 56 68 96 00 00 00 ff 90 c0 59 00 00 59 58 53 56 e8 bc fc ff ff 83 c4 10 85 c0 78 0d <89> 1e 85 f6 8d 46 04 75 a8 31 c0 eb a4 83 ec 08 53 56 e8 8f fe 
Sep 16 00:55:28 ev1s-xxx kernel:  <0>Fatal exception: panic in 5 seconds

I'm running Red Hat ES4 and MySQL 5.0.45 (Ver 14.12 Distrib 5.0.45, for pc-linux-gnu (i686) using readline 5.0) and PHP 5.2.2 (PHP scripts generate most of the queries).

Typical output of 'mysqladmin status':
Uptime: 1276  Threads: 256  Questions: 597441  Slow queries: 1  Opens: 564816  Flush tables: 1  Open tables: 137  Queries per second avg: 468.214

I'm including my.cnf and queries extracted from MySQL query log which were processed at the time of crash. I picked logs for this specific crash because of strange null bytes appearing in the log file. The MySQL error log doesn't contain any useful information.

Any help would be greatly appreciated.

How to repeat:
No special steps required.

my.cnf

Attachment: my.cnf (application/octet-stream, text), 1.48 KiB.

mysql query log

Attachment: mysql-query-crash.log (application/octet-stream, text), 36.98 KiB.

Thank you for a detailed problem report. Please, send also the results of:

uname -a
free

Linux commands.

Command results:

# uname -a
Linux xxx 2.6.9-55.0.6.ELsmp #1 SMP Thu Aug 23 11:11:20 EDT 2007 i686 i686 i386 GNU/Linux

# free
             total       used       free     shared    buffers     cached
Mem:       2074844    1509248     565596          0     261516     907184
-/+ buffers/cache:     340548    1734296
Swap:      4144748          0    4144748

Same issue here:

[root@xxxxx ~]# uname -a
Linux xxxxx 2.6.9-5.ELsmp #1 SMP Wed Jan 5 19:30:39 EST 2005 i686 i686 i386 GNU/Linux

[root@xxxxx ~]# cat /etc/redhat-release 
Red Hat Enterprise Linux AS release 4 (Nahant)

We upgraded yesterday from 5.0.28 to 5.0.45. We haven´t had any issue with 5.0.28 and now in 12 hours 2 kernel panics like this, so we thought that it was a hardware problem, memory related and the machine is now outside the production enviroment.

We have another 4 machines with 5.0.45, but fresh installation(this is the only difference), not upgrades without any issue doing exactly the same work, slaves , only myisam tables, 4 Gb database...

[root@xxxxx ~]# rpm -qa | grep MyS     
MySQL-shared-community-5.0.45-0.rhel4
MySQL-client-community-5.0.45-0.rhel4
MySQL-server-community-5.0.45-0.rhel4
MySQL-devel-community-5.0.45-0.rhel4

In a few days we will go to the datacenter to make some memory checks, to ensure this is a hardware problem or not.

Best regards,

System memory is Ok.

Checked with memtest86-3.3-1

Please, try to repeat with a newer version, 5.0.51, and inform about the results.

This is a Linux kernel bug. First you should try to upgrade your kernel to the latest version available from Red Hat and if it doesn't solve your problem, you should report the problem in the Red Hat bugzilla.

The subject systems' uptime is now 15 days. No kernel, MySQL or any other system updates that I am aware of were made. Seems like one of the tables used by a hosted site was causing the crashes. Still, I am unable to pursue this problem any longer. Today I have upgraded the kernel to 2.6.9-67.ELsmp and I'm sticking to MySQL 5.0.45 for the time being to see if something comes up.

Please, inform about any results with this new kernel. Send SHOW CREATE TABLE for that problematic table.

No feedback was provided for this bug for over a month, so it is
being suspended automatically. If you are able to provide the
information that was originally requested, please do so and change
the status of the bug back to "Open".