Description:
Greetings.
A few days ago MySQL started to crash minute by minute. The following errors are logged when it occurs:
########################### Start of logs ######################################
130716 9:26:32 [Note] /usr/sbin/mysqld: ready for connections.
Version: '5.0.51a-3ubuntu5.4-log' socket: '/var/run/mysqld/mysqld.sock' port: 3306 (Ubuntu)
130716 9:26:32InnoDB: Assertion failure in thread 2973969296 in file btr0cur.c line 160
InnoDB: Failing assertion: page_is_comp(get_page) == page_is_comp(page)
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: http://dev.mysql.com/doc/refman/5.0/en/forcing-recovery.html
InnoDB: about forcing recovery.
130716 9:26:32 - mysqld got signal 11;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help diagnose
the problem, but since we have already crashed, something is definitely wrong
and this may fail.
key_buffer_size=327155712
read_buffer_size=131072
max_used_connections=1
max_connections=10000
threads_connected=0
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_connections = 3450801 K
bytes of memory
Hope that's ok; if not, decrease some variables in the equation.
thd=(nil)
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
Cannot determine thread, fp=0xb1430fc8, backtrace may not be correct.
Stack range sanity check OK, backtrace follows:
0x81f22d3
0x839fb8a
0x83a0b0f
0x83ab815
0x837208e
0x8372855
0x8373948
0x835690a
0x83b6cfc
0x8339958
0xb7f344fb
0xb7d48e5e
New value of fp=(nil) failed sanity check, terminating stack trace!
Please read http://dev.mysql.com/doc/mysql/en/using-stack-trace.html and follow instructions on how to resolve the stack trace. Resolved
stack trace is much more helpful in diagnosing the problem, so please do
resolve it
The manual page at http://www.mysql.com/doc/en/Crashing.html contains
information that should help you find out what is causing the crash.
InnoDB: Log scan progressed past the checkpoint lsn 20 1567401069
130716 9:26:32 InnoDB: Database was not shut down normally!
InnoDB: Starting crash recovery.
InnoDB: Reading tablespace information from the .ibd files...
InnoDB: Restoring possible half-written data pages from the doublewrite
InnoDB: buffer...
InnoDB: Doing recovery: scanned up to log sequence number 20 1567961421
130716 9:26:33 InnoDB: Starting an apply batch of log records to the database...
InnoDB: Progress in percents: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99
InnoDB: Apply batch completed
130716 9:26:33 InnoDB: Started; log sequence number 20 1567961421
130716 9:26:33 [Warning] Found invalid password for user: 'auxiliadorafw@localhost'; Ignoring user
130716 9:26:33 [Warning] Found invalid password for user: 'agrobella@%'; Ignoring user
130716 9:26:33 [Warning] Found invalid password for user: 'sergiom@localhost'; Ignoring user
130716 9:26:33 [Warning] Found invalid password for user: 'filidesign@localhost'; Ignoring user
130716 9:26:33 [Warning] Found invalid password for user: 'mabella@localhost'; Ignoring user
130716 9:26:33 [Warning] Found invalid password for user: 'oprecodobarato@localhost'; Ignoring user
130716 9:26:33 [Warning] Found invalid password for user: 'menuzzi@localhost'; Ignoring user
130716 9:26:33 [Warning] Found invalid password for user: 'radiocaibi@localhost'; Ignoring user
130716 9:26:33 [Warning] Found invalid password for user: 'saojorgeturismo@localhost'; Ignoring user
130716 9:26:33 [Warning] Found invalid password for user: 'valitur@'; Ignoring user
130716 9:26:33 [Warning] Found invalid password for user: 'ivaregina@localhost'; Ignoring user
130716 9:26:33 [Warning] Found invalid password for user: 'agrobellaaliment@'; Ignoring user
130716 9:26:33 [Warning] Found invalid password for user: 'laurochielle@'; Ignoring user
130716 9:26:33 [Warning] Found invalid password for user: 'intracare@'; Ignoring user
130716 9:26:33 [Warning] Found invalid password for user: 'espacobambini@%'; Ignoring user
130716 9:26:33 [Warning] Found invalid password for user: 'f3publicidade@localhost'; Ignoring user
130716 9:26:33 [Warning] Found invalid password for user: 'liderrs@localhost'; Ignoring user
130716 9:26:33 [Warning] Found invalid password for user: 'excellenceengenh@localhost'; Ignoring user
130716 9:26:33 [Warning] Found invalid password for user: 'fredericostreetd@localhost'; Ignoring user
130716 9:26:33 [Warning] Found invalid password for user: 'radiovistaalegre@localhost'; Ignoring user
130716 9:26:33 [Warning] Found invalid password for user: 'impactodesp@localhost'; Ignoring user
130716 9:26:33 [Warning] Found invalid password for user: 'rlme@localhost'; Ignoring user
130716 9:26:33 [Warning] Neither --relay-log nor --relay-log-index were used; so replication may break when this MySQL server acts as a slave and has his hostname changed!! Please use '--relay-log=mysqld-relay-bin' to avoid this problem.
130716 9:26:34 [Note] /usr/sbin/mysqld: ready for connections.
################################################################################
The crash occurs os interval about 1 minute. Sometimes it reports some tables as crashed, but I dont' think its the source of the problem.
How to repeat:
I think this problem can be repeated only on our environment.
