MySQL Bugs: #69914: mysqlreplicate utility crashes freshly installed MySQL 5.6.12 slave servers

Bug #69914	mysqlreplicate utility crashes freshly installed MySQL 5.6.12 slave servers
Submitted:	2 Aug 2013 22:49	Modified:	1 Apr 2014 8:03
Reporter:	Shahriyar Rzayev	Email Updates:
Status:	Duplicate	Impact on me:	None
Category:	MySQL Utilities	Severity:	S1 (Critical)
Version:	5.6.12	OS:	Linux (Centos 6.4)
Assigned to:		CPU Architecture:	Any

Description:
This bug report is primarily related to my other 2 reports:
http://bugs.mysql.com/bug.php?id=69898
http://bugs.mysql.com/bug.php?id=69907

I tested 2 times with different machines that after several time running mysqlreplicate it crashes slave servers:

[root@localhost ~]# mysqlreplicate --master=root:12345@localhost:3306   --slave=remote:12345@192.168.1.5:3306 --rpl-user=repl:12345 -vv
# master on localhost: ... connected.
# slave on 192.168.1.5: ... connected.
# master id = 1
#  slave id = 2
# master uuid = 4fbde670-f4e3-11e2-ba65-2089846422ad
#  slave uuid = 8917b828-f6fc-11e2-8815-080027d5492c
# Checking InnoDB statistics for type and version conflicts.
# Checking storage engines...
# Checking for binary logging on master...
# Setting up replication...
# Connecting slave to master...
# CHANGE MASTER TO MASTER_HOST = 'localhost', MASTER_USER = 'repl', MASTER_PASSWORD = '12345', MASTER_PORT = 3306, MASTER_AUTO_POSITION=1
# Starting slave from master's last position...
# IO status: Connecting to master
# IO thread running: Connecting
# IO error: 1045:error connecting to master 'repl@localhost:3306' - retry-time: 60  retries: 1
# SQL thread running: Yes
# SQL error: None
# Waiting for slave to synchronize with master
# IO status: Connecting to master
# IO thread running: Connecting
# IO error: 1045:error connecting to master 'repl@localhost:3306' - retry-time: 60  retries: 1
# SQL thread running: Yes
# SQL error: None
# Waiting for slave to synchronize with master
# IO status: Connecting to master
# IO thread running: Connecting
# IO error: 1045:error connecting to master 'repl@localhost:3306' - retry-time: 60  retries: 1
.
.
# Waiting for slave to synchronize with master
ERROR: failed to synch slave with master.
ERROR: Cannot setup replication.

Output from error log of SLAVE server:

InnoDB: 2 transaction(s) which must be rolled back or cleaned up
InnoDB: in total 4 row operations to undo
InnoDB: Trx id counter is 10240
2013-08-03 11:21:47 7fc7f339d720  InnoDB: Error: a record lock wait happens in a dictionary operation!
InnoDB: index "CLUST_IND" of table "SYS_INDEXES".
InnoDB: Submit a detailed bug report to http://bugs.mysql.com
2013-08-03 11:21:47 7fc7f339d720  InnoDB: Assertion failure in thread 140496755873568 in file lock0wait.cc line 297
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: http://dev.mysql.com/doc/refman/5.6/en/forcing-innodb-recovery.html
InnoDB: about forcing recovery.
06:21:47 UTC - mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed, 
something is definitely wrong and this may fail.

key_buffer_size=8388608
read_buffer_size=131072
max_used_connections=0
max_threads=1
thread_count=0
connection_count=0
It is possible that mysqld could use up to 
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 8589 K  bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0x0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0 thread_stack 0x40000
/usr/sbin/mysqld(my_print_stacktrace+0x35)[0x8c70d5]
/usr/sbin/mysqld(handle_fatal_signal+0x4a4)[0x65cbe4]
/lib64/libpthread.so.0(+0xf500)[0x7fc7f498a500]
/lib64/libc.so.6(gsignal+0x35)[0x7fc7f36358a5]
/lib64/libc.so.6(abort+0x175)[0x7fc7f3637085]
/usr/sbin/mysqld[0x9171c5]
/usr/sbin/mysqld[0x94bd64]
/usr/sbin/mysqld[0x94c225]
/usr/sbin/mysqld[0x963d97]
/usr/sbin/mysqld[0x91d1b5]
/usr/sbin/mysqld[0x99aad4]
/usr/sbin/mysqld[0x8e311d]
/usr/sbin/mysqld(_Z24ha_initialize_handlertonP13st_plugin_int+0x48)[0x5a36c8]
/usr/sbin/mysqld[0x6e13c1]
/usr/sbin/mysqld(_Z11plugin_initPiPPci+0xb16)[0x6e4f76]
/usr/sbin/mysqld[0x596921]
/usr/sbin/mysqld(_Z11mysqld_mainiPPc+0x405)[0x59b4a5]
/lib64/libc.so.6(__libc_start_main+0xfd)[0x7fc7f3621cdd]
/usr/sbin/mysqld[0x58d701]
The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you find out what is causing the crash.
Writing a core file
130803 11:21:48 mysqld_safe mysqld from pid file /var/lib/mysql/localhost.localdomain.pid ended

Will attach core dump too.

How to repeat:
Installed MySQL 5.6.12 on Virtualbox(Centos 6.4) and tried setup GTID based replication with mysqlreplicate. Several time run it with wrong --repl-user and got a crash on SLAVE server.
Already reported.
But i didn't know mysqlreplicate causes this crash until now, as i tested it on 2 different laptops.

core dump of MySQL 5.6.12 crash

Attachment: core4.tar.gz (application/gzip, text), 2.19 MiB.

Can you attach more of the slave error log output?
The log that you attached is issued after server restart. What happened before that? Was the server killed, or did it crash by itself? I would like to have the full log from the previous startup up to these lines (issued by crash recovery):

InnoDB: 2 transaction(s) which must be rolled back or cleaned up
InnoDB: in total 4 row operations to undo
InnoDB: Trx id counter is 10240

Is this the same scenario as Bug#69898? Is it the same crashing instance too? (The files could be slightly different, because InnoDB is multi-threaded.)

Marko Mäkelä,

Yes it is the same instance and same scenario. and i think the cause of crashs is "mysqlreplicate"...
Thats why i report this as seperate bug.
Please repeat this scenario with mysqlreplicate.

Sadly i removed by accident the previous error log which maybe include information before server first crash :(
But you can test and repeat this scenario by your own. i think it must be repeatable by all

Sorry i must correct that in this report i used my different laptop but with same scenario. So it is different instance with same configuration and with same errors.
Thanks.

The mysqlreplicate utility is unlikely the culprit because it simply issues the same SQL commands you would run yourself for setting up replication. We need to know for certain it is the utility.

Please try to reproduce this problem without using mysqlreplicate. You should use the same CHANGE MASTER command as shown in the output of mysqlreplicate. More specifically, these commands (corrected as needed):

<on master>
GRANT REPLICATION SLAVE ON *.* TO 'repl'@'%' IDENTIFIED BY '12345';

<on slave>
CHANGE MASTER TO MASTER_HOST = 'localhost', MASTER_USER = 'repl', MASTER_PASSWORD = '12345', MASTER_PORT = 3306, MASTER_AUTO_POSITION=1;
START SLAVE;

If this fails, please post your results including the error log for the slave as well as the master.

If you cannot reproduce the problem manually, but can with mysqlreplicate, please include -vvv in the options and post the results.

Lastly, what version of MySQL Utilities are you using?

Charles Bell,
Due to another BUG report i have already crashed MySQL...because of crash i cant even start my slave. So i cant setup replication manually at this moment. 
So i can only attach all things that will be usefull to you repeat this crash with mysqlreplicate. All information you asked, i have already stated in these files:
master_server.txt --- here i stated all steps in master server
slave_server.txt --- here i stated all steps in slave server
replication_steps.txt --- here i stated all steps for replication setup.
slave_error_log
master_error_log

And i attached this files in this bug too:
http://bugs.mysql.com/bug.php?id=69898

All files with usefull information

Attachment: bug_attach.tar.gz (application/gzip, text), 9.27 KiB.

I have tried to reproduce this problem as you describe but I have been unable to. We really do need to know if a manual setup produces the same error on your systems. Once we know this, we can work toward trying to reproduce the problem more.

One last request. Please state how you setup your VM for the Centos OS (what host, etc., any special firewall or other configs, etc.) so that we can get as close to your configuration as possible.

I must inform you that all 2 related BUGS are verified:

http://bugs.mysql.com/bug.php?id=69898
http://bugs.mysql.com/bug.php?id=69898

"Please state how you setup your VM for the Centos OS (what host, etc., any special firewall or other configs, etc.)"
No special procedures just default Linux installation even i stopped iptables.

I will install new Centos in different VM and will test manually this issue as you say. 
But in another BUG especially in http://bugs.mysql.com/bug.php?id=69898 they can repeat my scenario.

Maybe this report is not important now, because Related BUG reports which i provide closed and fixed now with recent version of MySQL.

Is there any info that i must provide?
Or could yo reproduce same conditions?

Hi Shahriyar,

I think this bug should just be closed.  It was the result of a server bug #69898 .   At least, it will be useful to those who search google for similar problems if they are still using old server versions.