MySQL Bugs: #65831: mysql-5.5.25a crash

Bug #65831	mysql-5.5.25a crash
Submitted:	5 Jul 2012 21:18	Modified:	2 Nov 2012 15:36
Reporter:	Jean-Christophe Petit	Email Updates:
Status:	Duplicate	Impact on me:	None
Category:	MySQL Server: Replication	Severity:	S1 (Critical)
Version:	5.5.25a	OS:	Linux (RedHat-6.2 x86_64)
Assigned to:		CPU Architecture:	Any
Tags:	crash when enabling replication (START SLAVE)

Description:
Crash when doing "start slave;"

120705 17:07:00 mysqld_safe mysqld from pid file /var/lib/mysql/mysqld.pid ended
120705 17:07:11 mysqld_safe Starting mysqld daemon with databases from /mysqldata/var/lib/mysql
120705 17:07:11 [Note] Plugin 'FEDERATED' is disabled.
120705 17:07:11 [Note] Plugin 'BLACKHOLE' is disabled.
120705 17:07:11 InnoDB: The InnoDB memory heap is disabled
120705 17:07:11 InnoDB: Mutexes and rw_locks use GCC atomic builtins
120705 17:07:11 InnoDB: Compressed tables use zlib 1.2.3
120705 17:07:11 InnoDB: Using Linux native AIO
120705 17:07:11 InnoDB: Initializing buffer pool, size = 16.0G
120705 17:07:12 InnoDB: Completed initialization of buffer pool
120705 17:07:12 InnoDB: highest supported file format is Barracuda.
120705 17:07:13  InnoDB: Waiting for the background threads to start
120705 17:07:14 InnoDB: 1.1.8 started; log sequence number 171942364362
120705 17:07:14 [Note] Server hostname (bind-address): '0.0.0.0'; port: 3306
120705 17:07:14 [Note]   - '0.0.0.0' resolves to '0.0.0.0';
120705 17:07:14 [Note] Server socket created on IP: '0.0.0.0'.
120705 17:07:14 [Warning] 'proxies_priv' entry '@ root@mysqls53' ignored in --skip-name-resolve mode.
120705 17:07:14 [Note] Event Scheduler: Loaded 0 events
120705 17:07:14 [Note] /usr/sbin/mysqld: ready for connections.
Version: '5.5.25a-log'  socket: '/var/lib/mysql/mysqld.sock'  port: 3306  MySQL Community Server (GPL)
120705 17:07:24 [Note] Slave SQL thread initialized, starting replication in log 'mysql-bin.562782' at position 842909, relay log '/mysqldata/var/lib/mysql/mysqld-relay-bin.048945' position: 1165
120705 17:07:24 [Note] Slave I/O thread: connected to master 'remote@192.168.0.200:3306',replication started in log 'mysql-bin.562782' at position 1037074
21:07:24 UTC - mysqld got signal 11 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.

key_buffer_size=536870912
read_buffer_size=2097152
max_used_connections=2
max_threads=4096
thread_count=1
connection_count=1
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 17348288 K  bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0x7fc1d4000990
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 7fc6642fb838 thread_stack 0x40000
/usr/sbin/mysqld(my_print_stacktrace+0x35)[0x7a3f15]
/usr/sbin/mysqld(handle_fatal_signal+0x4a4)[0x67ddb4]
/lib64/libpthread.so.0(+0xf4a0)[0x7fc6695ef4a0]
/usr/sbin/mysqld(_Z12mysql_insertP3THDP10TABLE_LISTR4ListI4ItemERS3_IS5_ES6_S6_15enum_duplicatesb+0x274)[0x574c74]
/usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x885)[0x584ac5]
/usr/sbin/mysqld(_Z11mysql_parseP3THDPcjP12Parser_state+0x188)[0x588ca8]
/usr/sbin/mysqld(_ZN15Query_log_event14do_apply_eventEPK14Relay_log_infoPKcj+0xce8)[0x7250d8]
/usr/sbin/mysqld(_Z26apply_event_and_update_posP9Log_eventP3THDP14Relay_log_info+0x12d)[0x526f6d]
/usr/sbin/mysqld[0x52bd32]
/usr/sbin/mysqld(handle_slave_sql+0x9a1)[0x52d1f1]
/lib64/libpthread.so.0(+0x77f1)[0x7fc6695e77f1]
/lib64/libc.so.6(clone+0x6d)[0x7fc66886dccd]

Trying to get some variables.
Some pointers may be invalid and cause the dump to abort.
Query (7fc1d40156f6): is an invalid pointer
Connection ID (thread ID): 13
Status: NOT_KILLED

The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you find out what is causing the crash.

How to repeat:
Just doing "start slave;"

Going back to 5.5.24 "fix" the problem.

with query_cache disabled we have the same problem

With RedHat-5.5 x86_64 the problem does not seem to happen
Difference is:
120705 15:12:21 InnoDB: Using Linux native AIO
120705 15:12:21 InnoDB: Error: Linux Native AIO is not supported on tmpdir.
InnoDB: You can either move tmpdir to a file system that supports native AIO
InnoDB: or you can set innodb_use_native_aio to FALSE to avoid this message.
120705 15:12:21 InnoDB: Error: Linux Native AIO check on tmpdir returned error[22]

same problem with
innodb_use_native_aio=0

Please, send my.cnf file content from this slave. Can you try to identify crashing statement from the binary logs?

I wonder if this is related to bug #65740.

Hello,

in fact, it looks like an update is causing MySQL to crash, here are the last queries before the crash:

120706 22:59:35   433 Query     start slave
                  438 Query     BEGIN
                  437 Connect Out       remote@xxx.xxx.xxx.xxx:3306
                  439 Connect   root@localhost on
                  439 Query     select @@version_comment limit 1
                  439 Query     SHOW FULL PROCESSLIST
                  439 Quit
                  438 Query     UPDATE ..................................

I uploaded the my.cnf also as requested.

Have a good day

just to confirm that the UPDATE is similar to the one in bug #65740
update test set rank=@rnk:=@rnk+1 order by lat;

Even if it is more like:
update test set a='xxx', b='yyy', rank=@rnk:=rank WHERE d = 12345 AND e = 67890;

Jean-Christophe

Let's call it a duplicate of Bug #65740 then. 5.5.24 should work for you in the meantime.

We are having the same issue running RHEL 5.6 x86_64. Getting the same errors when starting slave replication and crashes mysql.

This bug has been marked as a duplicate of another, but that other bug is apparently restricted access / private. This is unhelpful :-/

I don't know why that bug is marked private. I've requested to find out why.

From Bug #65740 - Fixed in 5.1.65/5.5.27.

I still do not know why it is marked as private.

I do not believe this is fixed - and there is no way to confirm - 5.5.27 changelogs do not reference this bug nor the private bug in any way.

5.5.27 server that just crashed this morning gave this traceback:

Thread pointer: 0x7f63a4077460
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 7f6998a18758 thread_stack 0x40000
/usr/libexec/mysqld(my_print_stacktrace+0x2e)[0x78461e]
/usr/libexec/mysqld(handle_fatal_signal+0x493)[0x66f373]
/lib64/libpthread.so.0[0x3eb940f490]
/usr/libexec/mysqld(_Z11open_tablesP3THDPP10TABLE_LISTPjjP19Prelocking_strategy+0x575)[0x550d95]
/usr/libexec/mysqld(_Z12mysql_updateP3THDP10TABLE_LISTR4ListI4ItemES6_PS4_jP8st_ordery15enum_duplicatesbPySB_+0x10a)[0x5e96da]
/usr/libexec/mysqld(_Z21mysql_execute_commandP3THD+0x1bf8)[0x580e88]
/usr/libexec/mysqld(_Z11mysql_parseP3THDPcjP12Parser_state+0x1bb)[0x5839eb]
/usr/libexec/mysqld(_ZN15Query_log_event14do_apply_eventEPK14Relay_log_infoPKcj+0xae4)[0x70e864]
/usr/libexec/mysqld(_Z26apply_event_and_update_posP9Log_eventP3THDP14Relay_log_info+0x135)[0x5280a5]
/usr/libexec/mysqld[0x52cb0b]
/usr/libexec/mysqld(handle_slave_sql+0x946)[0x52dee6]
/lib64/libpthread.so.0[0x3eb94077e1]
/lib64/libc.so.6(clone+0x6d)[0x3eb90e577d]

We are aware that there is no mention in the changelog.  Please see this page:

http://ronaldbradford.com/blog/when-is-a-crashing-mysql-bug-not-a-bug-2012-08-15/

I do not see any my.cnf here. All reporters, please, check if you use any replicate-* options in my.cnf.

Jean-Christophe Petit,

I jsut checked test case for bug #65740 and it does not crash in 5.5.27 anymore. Please upgrade and test, so we can be sure your case is solved as well.

Andrew,

your case seems to be different from Bug #65740. I think you need to open new report, so we can start collecting information necessary to repeat your case.

Sveta,

You have a reproducible test case on #66890. It affects 5.5.27.

Hello,

bug seems to be fixed in MySQL-5.5.28
We've been running a slave for a more than a week now.

JC