MySQL Bugs: #64505: MySQL Slave crashes upon restart

Bug #64505	MySQL Slave crashes upon restart
Submitted:	1 Mar 2012 6:33	Modified:	23 Mar 2012 9:23
Reporter:	Emmanuel KARTMANN	Email Updates:
Status:	Duplicate	Impact on me:	None
Category:	MySQL Server: Replication	Severity:	S1 (Critical)
Version:	5.5.21	OS:	Windows (Windows Server 2008 R2)
Assigned to:		CPU Architecture:	Any
Tags:	crash, slave, start

Description:
One of our replication slaves is crashing upon every restart - this occurs every night (we restart the slave automatically to create a zip file from all databases).

The error log contains:

120301  0:01:08 [ERROR] Error reading packet from server: Lost connection to MySQL server during query ( server_errno=2013)
120301  0:01:08 [Note] Slave I/O thread exiting, read up to log 'mysql-bin.000265', position 15103964
120301  0:01:08 [Note] C:\PROGRA~1\MySQL\MYSQLS~1.5\bin\mysqld.exe: Normal shutdown

120301  0:01:08 [Note] Event Scheduler: Purging the queue. 0 events
120301  0:01:08 [Note] C:\PROGRA~1\MySQL\MYSQLS~1.5\bin\mysqld.exe: Shutdown complete

120301  0:04:23 [Note] Slave I/O thread: connected to master 'lutecia_backup@lutecia-db:3306',replication started in log 'mysql-bin.000265' at position 15103964
120301  0:04:23 [Note] Event Scheduler: Loaded 0 events
120301  0:04:23 [Note] C:\PROGRA~1\MySQL\MYSQLS~1.5\bin\mysqld.exe: ready for connections.
Version: '5.5.16-log'  socket: ''  port: 3306  MySQL Community Server (GPL)
120301  3:15:11 - mysqld got exception 0xc0000005 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help diagnose
the problem, but since we have already crashed, something is definitely wrong
and this may fail.

key_buffer_size=134217728
read_buffer_size=131072
max_used_connections=1
max_threads=350
thread_count=0
connection_count=0
It is possible that mysqld could use up to 
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 3259648 K
bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0xbc73f00
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
000000013FE0C5B2    mysqld.exe!?compatible_with@table_def@@QEBA_NPEAVTHD@@PEAVRelay_log_info@@PEAUTABLE@@PEAPEAU4@@Z()
000000013FD33034    mysqld.exe!?do_apply_event@Rows_log_event@@EEAAHPEBVRelay_log_info@@@Z()
000000013FC500EC    mysqld.exe!?apply_event_and_update_pos@@YAHPEAVLog_event@@PEAVTHD@@PEAVRelay_log_info@@@Z()
000000013FC542E3    mysqld.exe!?show_master_info@@YA_NPEAVTHD@@PEAVMaster_info@@@Z()
000000013FC5593A    mysqld.exe!handle_slave_sql()
000000013FE3201E    mysqld.exe!win_pthread_mutex_trylock()
000000013FFD3AA7    mysqld.exe!my_mb_ctype_mb()
000000013FFD3B5B    mysqld.exe!my_mb_ctype_mb()
000000007714652D    kernel32.dll!BaseThreadInitThunk()
000000007727C521    ntdll.dll!RtlUserThreadStart()

Trying to get some variables.
Some pointers may be invalid and cause the dump to abort.
Query (0000000000000000): 
Connection ID (thread ID): 1
Status: NOT_KILLED

The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you find out what is causing the crash.

How to repeat:
Happens every night - on this particular server...

Suggested fix:
None!

Please, check if the same problem still happens with a newer version on slave, 5.5.21.

Ok I will check if crash occurs with latest version, but this could take some time, since it's a production system and we need to test version 5.5.21 before upgrading...

After upgrade to 5.5.21, the SLAVE has crashed tonight.

Any hint on how to debug/trace this crash?

E.

Can you try to get the exact statement (from master's binary log and/or from the relay log) that slave was processing when crashed? Had you got the same stack trace with 5.5.21 as with 5.5.16?

Please, send my.ini file content from both master and slave. Attach also the entire error log from slave, compressed, if possible.

SLAVE: my.ini, binary log, error log

Attachment: SLAVE.7z (application/octet-stream, text), 2.91 KiB.

I attached the SLAVE information, but even when the slave crashed, we had no explicit "crash" in the error log (?).
Note that the crash occurs when we automatically restart the slave, when there may be some replication activity going on (thus the crash is not occuring 100% of the time).

We upgraded SLAVE first to version 5.5.21 - it crashed.
We then upgraded MASTER to version 5.5.21 - slave did crashed.

Thank you for the feedback.

How do you automatically restart the slave? Which command do you use?

A scheduled task does stop the slave in two steps:
 mysqladmin.exe shutdown
 net stop mysql

If the first command fails, it executes the second.
Then the scheduled task rotate the log files (on windows we can't rotate logs while MySQL is running...).

Thank you for the feedback.

> A scheduled task does stop the slave in two steps:
> mysqladmin.exe shutdown
> net stop mysql
>
>If the first command fails, it executes the second.

Interesting why mysqladmin fails, but net stop should not crash server anyway.

> Then the scheduled task rotate the log files (on windows we can't rotate logs while MySQL is running...).

Do you mean relay log files? If so how do you rotate them?

Have you tried to run STOP SLAVE before shutting down MySQL server?

I got it : we have hit Bug #61450 (Fix for bug #47103 have not been backported into 5.5 series). We have a merge table that get updated every night, and it crashes the MySQL SLAVE (if we don't update that particular table, there IS NO CRASH). 

You can mark this bug as a replicate of #61450.

Duplicate of Bug #61450.