Bug #68048 Mysql crashed
Submitted: 7 Jan 2013 13:26 Modified: 13 Feb 2013 15:37
Reporter: Amit Anand Email Updates:
Status: No Feedback Impact on me:
None 
Category:MySQL Server: InnoDB storage engine Severity:S3 (Non-critical)
Version:5.0.90-enterprise OS:Linux
Assigned to: CPU Architecture:Any

[7 Jan 2013 13:26] Amit Anand
Description:
Mysql crahed with following errors.

Version: '5.0.90-enterprise'  socket: '/var/lib/mysql/mysql.sock'  port: 3306  MySQL Enterprise Server (Commercial)
InnoDB: Database page corruption on disk or a failed
InnoDB: file read of page 10761717.
InnoDB: You may have to recover from a backup.
121230 15:21:37  InnoDB: Page dump in ascii and hex (16384 bytes):

;InnoDB: End of page dump
121230 15:21:37  InnoDB: Page checksum 2400682541, prior-to-4.0.14-form checksum 3724978503
InnoDB: stored checksum 1497199900, prior-to-4.0.14-form stored checksum 0
InnoDB: Page lsn 295 317918090, low 4 bytes of lsn at page end 0
InnoDB: Page number (if stored to page already) 10761717,
InnoDB: space id (if created with >= MySQL-4.1.1 and stored already) 0
InnoDB: Page may be an index page where index id is 0 433
InnoDB: (index PRIMARY of table bn/cdrsummarybymonth)
InnoDB: Database page corruption on disk or a failed
InnoDB: file read of page 10761717.
InnoDB: You may have to recover from a backup.
InnoDB: It is also possible that your operating
InnoDB: system has corrupted its own file cache
InnoDB: and rebooting your computer removes the
InnoDB: error.
InnoDB: If the corrupt page is an index page
InnoDB: you can also try to fix the corruption
InnoDB: by dumping, dropping, and reimporting
InnoDB: the corrupt table. You can use CHECK
InnoDB: TABLE to scan your table for corruption.
InnoDB: See also InnoDB: http://dev.mysql.com/doc/refman/5.0/en/forcing-recovery.html
InnoDB: about forcing recovery.
InnoDB: Ending processing because of a corrupt database page.

Number of processes running now: 0
121230 15:21:38  mysqld restarted
InnoDB: Log scan progressed past the checkpoint lsn 295 817937505
121230 15:21:40  InnoDB: Database was not shut down normally!
InnoDB: Starting crash recovery.
InnoDB: Reading tablespace information from the .ibd files..

How to repeat:
Not known
[7 Jan 2013 20:41] Shane Bester
Amit, oops, please check if /var/log/messages contain anything regarding disks, memory issues?  What was the last version of mysql these tables got rebuilt in?
[8 Jan 2013 12:44] Amit Anand
There is nothing in messages indicating about memory or disk space.

Dec 30 14:10:01 qic-rsm /usr/sbin/cron[4910]: (root) CMD ((cd /home/nextone/scripts; ./longquery.sh))
Dec 30 14:10:01 qic-rsm /usr/sbin/cron[4911]: (root) CMD ((cd /home/nextone/scripts; ./rsm_control.sh))
Dec 30 14:17:06 qic-rsm kernel: [10680494.322247] martian source 178.22.8.84 from 178.22.8.81, on dev eth1
Dec 30 14:17:06 qic-rsm kernel: [10680494.322254] ll header: ff:ff:ff:ff:ff:ff:00:10:db:ff:20:00:08:06
Dec 30 14:20:01 qic-rsm /usr/sbin/cron[5522]: (root) CMD ((cd /home/nextone/scripts; ./collect_stat_rsm.sh >> /var/log/rsm_stat.log))
Dec 30 14:20:01 qic-rsm /usr/sbin/cron[5523]: (root) CMD ((cd /home/nextone/scripts; ./longquery.sh))
Dec 30 14:30:01 qic-rsm /usr/sbin/cron[5933]: (root) CMD ((cd /home/nextone/scripts; ./longquery.sh))
Dec 30 14:30:02 qic-rsm syslog-ng[1917]: Log statistics; dropped='pipe(/dev/xconsole)=0', dropped='pipe(/dev/tty10)=0', processed='center(queued)=25164', processed='cen
ter(received)=21340', processed='destination(newsnotice)=0', processed='destination(acpid)=0', processed='destination(firewall)=17224', processed='destination(null)=0',
 processed='destination(mail)=2906', processed='destination(mailinfo)=2900', processed='destination(console)=292', processed='destination(newserr)=0', processed='destin
ation(newscrit)=0', processed='destination(messages)=1210', processed='destination(mailwarn)=0', processed='destination(localmessages)=0', processed='destination(netmgm
)=0', processed='destination(mailerr)=6', processed='destination(xconsole)=292', processed='destination(warn)=334', processed='source(src)=21340'
Dec 30 14:35:01 qic-rsm /usr/sbin/cron[6175]: (root) CMD ((cd /home/nextone/scripts; ./collect_stat_rsm.sh >> /var/log/rsm_stat.log))
Dec 30 14:40:01 qic-rsm /usr/sbin/cron[6376]: (root) CMD ((cd /home/nextone/scripts; ./longquery.sh))
Dec 30 14:50:01 qic-rsm /usr/sbin/cron[6820]: (root) CMD ((cd /home/nextone/scripts; ./collect_stat_rsm.sh >> /var/log/rsm_stat.log))
Dec 30 14:50:01 qic-rsm /usr/sbin/cron[6821]: (root) CMD ((cd /home/nextone/scripts; ./longquery.sh))
Dec 30 15:00:01 qic-rsm /usr/sbin/cron[7225]: (root) CMD ((cd /home/nextone/scripts; ./longquery.sh))
Dec 30 15:01:06 qic-rsm kernel: [10683134.421513] martian source 178.22.8.84 from 178.22.8.81, on dev eth1
Dec 30 15:01:06 qic-rsm kernel: [10683134.421519] ll header: ff:ff:ff:ff:ff:ff:00:10:db:ff:20:00:08:06
Dec 30 15:05:01 qic-rsm /usr/sbin/cron[7453]: (root) CMD ((cd /home/nextone/scripts; ./collect_stat_rsm.sh >> /var/log/rsm_stat.log))
Dec 30 15:07:44 qic-rsm sshd[7560]: pam_tally(sshd:auth): unknown option: no_magic_root
Dec 30 15:07:44 qic-rsm sshd[7560]: pam_tally(sshd:account): option deny=500 allowed in auth phase only
Dec 30 15:07:44 qic-rsm sshd[7560]: pam_tally(sshd:account): unknown option: no_magic_root
Dec 30 15:07:44 qic-rsm sshd[7560]: pam_tally(sshd:account): unknown option: reset
Dec 30 15:07:44 qic-rsm sshd[7560]: pam_tally(sshd:account): option per_user allowed in auth phase only
Dec 30 15:07:44 qic-rsm sshd[7558]: Accepted keyboard-interactive/pam for root from 85.153.8.2 port 21489 ssh2
Dec 30 15:07:44 qic-rsm sshd[7558]: pam_tally(sshd:setcred): unknown option: no_magic_root
Dec 30 15:07:45 qic-rsm sshd[7561]: pam_tally(sshd:setcred): unknown option: no_magic_root
Dec 30 15:10:01 qic-rsm /usr/sbin/cron[7879]: (root) CMD ((cd /home/nextone/scripts; ./longquery.sh))
Dec 30 15:10:01 qic-rsm /usr/sbin/cron[7880]: (root) CMD ((cd /home/nextone/scripts; ./rsm_control.sh))
Dec 30 15:20:01 qic-rsm /usr/sbin/cron[8886]: (root) CMD ((cd /home/nextone/scripts; ./collect_stat_rsm.sh >> /var/log/rsm_stat.log))
Dec 30 15:20:01 qic-rsm /usr/sbin/cron[8885]: (root) CMD ((cd /home/nextone/scripts; ./longquery.sh))
Dec 30 15:30:01 qic-rsm /usr/sbin/cron[9333]: (root) CMD ((cd /home/nextone/scripts; ./longquery.sh))
Dec 30 15:30:01 qic-rsm syslog-ng[1917]: Configuration reload request received, reloading configuration;
Dec 30 15:30:01 qic-rsm syslog-ng[1917]: New configuration initialized;
Dec 30 15:30:16 qic-rsm kernel: klogd 1.4.1, ---------- state change ----------
Dec 30 15:35:01 qic-rsm /usr/sbin/cron[9710]: (root) CMD ((cd /home/nextone/scripts; ./collect_stat_rsm.sh >> /var/log/rsm_stat.log))
Dec 30 15:40:01 qic-rsm /usr/sbin/cron[9896]: (root) CMD ((cd /home/nextone/scripts; ./longquery.sh))
Dec 30 15:50:01 qic-rsm /usr/sbin/cron[10302]: (root) CMD ((cd /home/nextone/scripts; ./collect_stat_rsm.sh >> /var/log/rsm_stat.log))
Dec 30 15:50:01 qic-rsm /usr/sbin/cron[10303]: (root) CMD ((cd /home/nextone/scripts; ./longquery.sh))
Dec 30 15:54:07 qic-rsm kernel: [10686315.540185] martian source 178.22.8.84 from 178.22.8.81, on dev eth1
Dec 30 15:54:07 qic-rsm kernel: [10686315.540190] ll header: ff:ff:ff:ff:ff:ff:00:10:db:ff:20:00:08:06
Dec 30 16:00:01 qic-rsm /usr/sbin/cron[10696]: (root) CMD ((cd /home/nextone/scripts; ./longquery.sh))

The mysql files were build in same version 5.0.90
[10 Jan 2013 9:44] Amit Anand
Hi Team,

Kindly suggest on this.

Thanks
Amit
[10 Jan 2013 9:50] Shane Bester
Hi!

Multiple pages/tables are corrupted according to the error log. You should probably try mysqldump of all innodb tables, and reload them into a cleanly recreated tablespace.

http://dev.mysql.com/doc/refman/5.0/en/forcing-innodb-recovery.html

If you have a backup, it could come in useful too.  You could also consider doing disk tests to check if there are any problems there.
[14 Feb 2013 1:00] Bugs System
No feedback was provided for this bug for over a month, so it is
being suspended automatically. If you are able to provide the
information that was originally requested, please do so and change
the status of the bug back to "Open".