Bug #74824 Mysql got crash with glibc errors
Submitted: 12 Nov 2014 21:50 Modified: 14 Dec 2014 22:50
Reporter: Sunil Divyeshvar Email Updates:
Status: No Feedback Impact on me:
None 
Category:MySQL Server Severity:S1 (Critical)
Version:Mysql-5.6.16 OS:Linux (Centos 6.4)
Assigned to: CPU Architecture:Any
Tags: crash, glibc detected

[12 Nov 2014 21:50] Sunil Divyeshvar
Description:
Hello, 

Mysql server-5.6.16 got crashed , i dont know the particular scenario so i am not able to regenrate that crash. 

I do have SAN storage connected with Mysql server. 

Please check following error log. 
First i think it might be some hardware or i/o issue but i got error like "*** glibc detected *** /usr/bin/mysqld: corrupted double-linked list: 0x00000000021f3290 ***" 

ErrorLog File= 

2014-11-09 07:32:50 42956 [ERROR] InnoDB: The OS said file flush did not succeed 
2014-11-09 07:32:50 42956 [ERROR] InnoDB: Tried to read 32768 bytes at offset 32028901376. Was only able to read -1. 
2014-11-09 07:32:50 42956 [ERROR] InnoDB: Tried to read 8192 bytes at offset 16622018560. Was only able to read -1. 
2014-11-09 07:32:50 7f118cdf2700 InnoDB: Operating system error number 5 in a file operation. 
InnoDB: Error number 5 means 'Input/output error'. 
2014-11-09 07:32:50 7f118ce6c700 InnoDB: Operating system error number 5 in a file operation. 
InnoDB: Some operating system error numbers are described at 
InnoDB: http://dev.mysql.com/doc/refman/5.6/en/operating-system-error-codes.html 
2014-11-09 07:32:50 7f135cdb07002014-11-09 07:32:50 42956 [ERROR] InnoDB: File (unknown): 'read' returned OS error 105. Cannot continue operation 
InnoDB: Operating system error number 5 in a file operation. 
InnoDB: Error number 5 means 'Input/output error'. 
InnoDB: Some operating system error numbers are described at 
InnoDB: http://dev.mysql.com/doc/refman/5.6/en/operating-system-error-codes.html 
InnoDB: Error number 5 means 'Input/output error'. 
2014-11-09 07:32:50 42956 [ERROR] InnoDB: File (unknown): 'read' returned OS error 105. Cannot continue operation 
InnoDB: Some operating system error numbers are described at 
InnoDB: http://dev.mysql.com/doc/refman/5.6/en/operating-system-error-codes.html 
2014-11-09 07:32:50 42956 [ERROR] InnoDB: File (unknown): 'flush' returned OS error 105. Cannot continue operation 
*** glibc detected *** /usr/bin/mysqld: corrupted double-linked list: 0x00000000021f3290 *** 
07:32:50 UTC - mysqld got signal 11 ; 
This could be because you hit a bug. It is also possible that this binary 
or one of the libraries it was linked against is corrupt, improperly built, 
or misconfigured. This error can also be caused by malfunctioning hardware. 
We will try our best to scrape up some info that will hopefully help 
diagnose the problem, but since we have already crashed, 
something is definitely wrong and this may fail. 

key_buffer_size=1073741824 
read_buffer_size=5242880 
max_used_connections=786 
max_threads=500 
thread_count=94 
connection_count=93 
It is possible that mysqld could use up to 
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 6182240 K bytes of memory 
Hope that's ok; if not, decrease some variables in the equation. 

Thread pointer: 0x0 
Attempting backtrace. You can use the following information to find out 
where mysqld died. If you see no messages after this, something went 
terribly wrong... 
pure virtual method called 
pure virtual method called 
terminate called without an active exception 
terminate called recursively 
pure virtual method called 
Fatal signal 6 while backtracing 
terminate called recursively 
141109 07:32:51 mysqld_safe Number of processes running now: 0 
141109 07:32:51 mysqld_safe mysqld restarted 

Please help

How to repeat:
i dont know the particular scenario so i am not able to regenrate that crash. 

Suggested fix:
Do not know about suggested fix , why it is telling "*** glibc detected *** /usr/bin/mysqld: corrupted double-linked list: 0x00000000021f3290 ***"
[13 Nov 2014 10:47] MySQL Verification Team
Please check your environment for the messages errors done by innoDB:

2014-11-09 07:32:50 42956 [ERROR] InnoDB: The OS said file flush did not succeed 
2014-11-09 07:32:50 42956 [ERROR] InnoDB: Tried to read 32768 bytes at offset 32028901376. Was only able to read -1. 
2014-11-09 07:32:50 42956 [ERROR] InnoDB: Tried to read 8192 bytes at offset 16622018560. Was only able to read -1. 
2014-11-09 07:32:50 7f118cdf2700 InnoDB: Operating system error number 5 in a file operation. 
InnoDB: Error number 5 means 'Input/output error'.
[13 Nov 2014 15:33] Sunil Divyeshvar
Yes i got that but what does following message mean ?

*** glibc detected *** /usr/bin/mysqld: corrupted double-linked list: 0x00000000021f3290 ***
[14 Nov 2014 22:50] MySQL Verification Team
Could mean hardware problems check your hard-drive etc.
[15 Dec 2014 1:00] Bugs System
No feedback was provided for this bug for over a month, so it is
being suspended automatically. If you are able to provide the
information that was originally requested, please do so and change
the status of the bug back to "Open".