Bug #45011 Mysql Crashes Every 2 minutes
Submitted: 21 May 2009 16:17 Modified: 25 May 2009 7:00
Reporter: Bill Cuevas Email Updates:
Status: Not a Bug Impact on me:
None 
Category:MySQL Server: Errors Severity:S1 (Critical)
Version:5.0.81-log OS:Linux
Assigned to: CPU Architecture:Any
Tags: crash, restart, thd->query at (nil)

[21 May 2009 16:17] Bill Cuevas
Description:
Our mysql server continues to crash and restart every 2 minutes.  I attempted use a stack trace but it did not return any results.
Below is the error I am receiving in the error logs.

090521  9:00:03  InnoDB: Started; log sequence number 103 3864215070
090521  9:00:03 [Note] Recovering after a crash using lease750-mysql-bin
090521  9:00:03 [Note] Starting crash recovery...
090521  9:00:03 [Note] Crash recovery finished.
090521  9:00:03 [Note] /usr/local/mysql/libexec/mysqld: ready for connections.
Version: '5.0.81-log'  socket: '/tmp/mysql.sock'  port: 3306  Source distribution
090521  9:02:03 - mysqld got signal 11 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help diagnose
the problem, but since we have already crashed, something is definitely wrong
and this may fail.

key_buffer_size=33554432
read_buffer_size=2097152
max_used_connections=21
max_connections=400
threads_connected=3
It is possible that mysqld could use up to 
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_connections = 1671168 K
bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

thd=0x2aa22a92f0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
Cannot determine thread, fp=0x2aa1a40d28, backtrace may not be correct.
Stack range sanity check OK, backtrace follows:
0x697300726573555a
Stack trace seems successful - bottom reached
Please read http://dev.mysql.com/doc/mysql/en/using-stack-trace.html and follow instructions on how to resolve the stack trace. Resolved
stack trace is much more helpful in diagnosing the problem, so please do 
resolve it
Trying to get some variables.
Some pointers may be invalid and cause the dump to abort...
thd->query at (nil)  is invalid pointer
thd->thread_id=742
The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you find out what is causing the crash.

Number of processes running now: 0
090521 09:02:03  mysqld restarted
InnoDB: Log scan progressed past the checkpoint lsn 103 3865318764
090521  9:02:03  InnoDB: Database was not shut down normally!
InnoDB: Starting crash recovery.
InnoDB: Reading tablespace information from the .ibd files...
InnoDB: Restoring possible half-written data pages from the doublewrite
InnoDB: buffer...
InnoDB: Doing recovery: scanned up to log sequence number 103 3865360080
090521  9:02:03  InnoDB: Starting an apply batch of log records to the database...
InnoDB: Progress in percents: 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 
InnoDB: Apply batch completed
InnoDB: Last MySQL binlog file position 0 956688, file name ./lease750-mysql-bin.000618
090521  9:02:04  InnoDB: Started; log sequence number 103 3865360080
090521  9:02:04 [Note] Recovering after a crash using lease750-mysql-bin
090521  9:02:04 [Note] Starting crash recovery...
090521  9:02:04 [Note] Crash recovery finished.
090521  9:02:04 [Note] /usr/local/mysql/libexec/mysqld: ready for connections.
Version: '5.0.81-log'  socket: '/tmp/mysql.sock'  port: 3306  Source distribution

How to repeat:
It continually happens every two minutes.
[22 May 2009 5:31] Sveta Smirnova
Thank you for the report.

Have you run mysql_upgrade after upgraded? If not, please so so.

Do you remember after which event this started to occur?

> thd->query at (nil)  is invalid pointer
> thd->thread_id=742

Shows this is not startup error, but some application connects to the server and queries it? Is it true? If so, you can try to find problem query using MySQL Proxy.

Also, please, use mysqld-debug binary and add option --core to your configuration file, so it generates core dump. Then upload core file to our FTP server. In this case we can get more information about the problem.
[22 May 2009 12:07] Bill Cuevas
Sveta,
I found the problem it was a remote java application. 

Thanks for you assistance.
[25 May 2009 7:00] Sveta Smirnova
Thank you for the feedback.

Closing as "Not a Bug" according to last comment.

Although it is not clear why mysqld crashes. So if you are able to create repeatable test case feel free to upload it and reopen the report.