MySQL Bugs: #32076: MySQL crash on ha_delete_hash_node --> handle

Bug #32076	MySQL crash on ha_delete_hash_node --> handle_segfault with InnoDB tables
Submitted:	3 Nov 2007 18:02	Modified:	18 Dec 2008 21:40
Reporter:	Scott Bailey	Email Updates:
Status:	No Feedback	Impact on me:	None
Category:	MySQL Server: InnoDB storage engine	Severity:	S1 (Critical)
Version:	5.0.27 - Standard	OS:	Linux (CentOS)
Assigned to:	Assigned Account	CPU Architecture:	Any
Tags:	corruption, crash, handle_segfault, innodb, random, SELECT, Signal 11

Description:
Hello, 

When making select queries against a mysql database, the database will often crash with a signal 11. Doing a stack trace reveals that the crash will always occur at while executing ha_delete_hash_node() --> handle_segfault()

Mysql will crash on select statements, event simple ones. For example: "select name from users where id = 5". Re-running the statement that causes it to crash will not cause it to crash a second time. None of the logs show a crash on an insert or update statement. 

The database has been scanned and shows no signs of corruption. 

I've included 3 crash entries from the mysqld.log (stack traces resolved). The common theme between all of them is that they crash while following the path ha_delete_hash_node --> handle_segfault()

Hope this information helps in tracking down a possible bug. 

Cheers!
Scott

How to repeat:
Unfortunatly I have been unable to consistently repeat this problem. 

What I have noticed, however, is that it is most likely to occur while a large series of queries are placed at once. 

For exmaple, I create a script that will run through the every entry in a user table and request data. 
Ie:

Select * from users where id = 1
Select * from users where id = 2
...

This will ussually cause a crash, though the crash will be caused on a different select statement every time. 

I have not been able to come up with a repeatible set of steps that can consistently cause the error though.

Error Log and Stack Trace

Attachment: error_log.txt (text/plain), 9.59 KiB.

Free

Attachment: free.txt (text/plain), 234 bytes.

Assigning to Inaam.

Scott,

Please attach the full error log.

regards,
inaam

No feedback was provided for this bug for over a month, so it is
being suspended automatically. If you are able to provide the
information that was originally requested, please do so and change
the status of the bug back to "Open".

Large Mysqld log

Attachment: mysqld_2.log (application/octet-stream, text), 127.02 KiB.

Hi Inaam, 

I've attachd a larger log file for review. 

You can also find a more complete(massive 22mb log file) at :

http://www.scribly.com/site/webroot/temp/

that contains more of the same. The same crashing pattern has continued for the last 7 months, with the database crashing about every 4-5 hours or so. Every crash is a different query and the same query can never crash the database again. All stack traces are always triggered at ha_delete_hash_node

Cheers, 
Scott

Scott,

Can you please upload the resolved stack trace?

http://dev.mysql.com/doc/refman/5.0/en/resolve-stack-dump.html

regards,
inaam

Scott,

OK. I see that you have provided resolved stack trace in your earlier notes. If it is the same stack trace each time, please disregard my previous note.

regards,
inaam

Hi Inaam, 

Thank you for your response. Yes, it is the same stack trace every time. 

Cheers, 
Scott

Scott,

we have found a bug in InnoDB's DROP TABLE / ALTER TABLE / CREATE INDEX that could cause the crash reported here.

Do you run such statements at the moment mysqld crashes?

Regards,

Heikki

Hi Heikki, 

Thank you for the update. 

I'm afraid that I'm not making any Alter/Drop/Create statements at the time of crash. In the case of this bug, it only occurs on SELECT statements. The interesting part of the problem is that it crashes on different SELECT statements every time - so its not the case of particular rows causing the problem. 

For example, calling "SELECT FROM tablename WHERE id='1'" may cause a crash one time, but calling it subsequently will not cause it to crash again. It occurs in different tables, but only in tables with more than 100,000 rows. The larger the table, the more likely that the select will die on it. All tables have been restored from a fresh backup before, but the problem continues - so I don't think table corruption is behind the problem. 

The crash usually happens under heavy load, where a lot of select statements are being done simultaneously (50-100 per second). The crash occurs every 3 hours or so, causing the mysql server to reboot. 

Thanks for your help!
Scott Bailey

Scott,

Wondering if you are still encountering this issue regularly. If so, is it the same stack each time? Is there anything new that you can share with us regarding this issue?

regards,
inaam

Setting to need feedback for last question from Inaam.

No feedback was provided for this bug for over a month, so it is
being suspended automatically. If you are able to provide the
information that was originally requested, please do so and change
the status of the bug back to "Open".