| Bug #67009 | MySQL upgrade from 5.5.11 to 5.5.25a in master and slave setup is a diaster! | ||
|---|---|---|---|
| Submitted: | 28 Sep 2012 16:28 | Modified: | 6 Dec 2012 20:06 |
| Reporter: | steven tang | Email Updates: | |
| Status: | Not a Bug | Impact on me: | |
| Category: | MySQL Server: Replication | Severity: | S1 (Critical) |
| Version: | 5.5.25a | OS: | Linux (master upgrade caused slave crash and had to be rebuilt) |
| Assigned to: | CPU Architecture: | Any | |
[28 Sep 2012 16:38]
steven tang
after master was upgraded from 5.5.11 to 5.5.25a, the slave crashed and complained about performance_schema problem. I copied master performance_schema to slave and replaced it with master one and restarted slave, then I got countless following errors: 120927 9:35:54 InnoDB: Error: page 517783 log sequence number 12555394483935 InnoDB: is in the future! Current system log sequence number 12552703479261. InnoDB: Your database may be corrupt or you may have copied the InnoDB InnoDB: tablespace but not the InnoDB log files. See InnoDB: http://dev.mysql.com/doc/refman/5.5/en/forcing-innodb-recovery.html InnoDB: for more information. there seems no solution for this problem from MySQL and internet. I had to rebuild slave by copy master (200GB).
[13 Oct 2012 17:59]
MySQL Verification Team
please provide the slaves error log. also please disable slave query_cache if it's enabled and report the results
[15 Oct 2012 15:43]
steven tang
I did not keep the error log as I had to rebuild the slave.
[15 Oct 2012 15:45]
steven tang
Could you please give me the detailed steps of upgrade a replaction setup MySQL servers of 5.5.11 to 5.5.XX? Is there anything wrong on what I had done? Thanks, Steven
[15 Oct 2012 15:54]
Peter Laursen
In my understanding you should upgrade the slave first. There is no guarantee that replication will work if slave server version is lower than master server version (but slave higher than master should be OK). Peter (not a MySQL/Oracle person)
[15 Oct 2012 16:34]
steven tang
Please check the description where I have listed my upgrade steps.
[15 Oct 2012 17:16]
MySQL Verification Team
Steven, you mentioned a crash. We need to know the nature of the crash (stack trace in error log). But I could predict it to be a known crash, hence I suggest disable query cache on slave. We really need the error logs.... Maybe setup a test instance and repeat it so we know for sure :)
[15 Oct 2012 17:27]
steven tang
I understand that and I think we have figured out that before we run mysql_upgrade on master, we need to change my.cnf to do following: 1. disable log-bin 2. skip-networking start master with above and then run mysql_upgrade. After that, stop master and change my.cnf to do following: 1. enable log-bin 2. enable networking and restart master and everything seems to work fine.
[18 Oct 2012 19:27]
MySQL Verification Team
It seems you could have used the --skip-write-binlog option to mysql_upgrade then? http://dev.mysql.com/doc/refman/5.5/en/mysql-upgrade.html#option_mysql_upgrade_write-binlo...
[6 Dec 2012 20:06]
Sveta Smirnova
Thank you for the update. Since you resolved the problem and there is actually description of proper mysql_upgrade option in our user manual I close this as "Not a bug"
Description: We followed the MySQL 5.5 upgrade document on upgrading slave from 5.5.11 to 5.5.25a first 2 months ago, and finally we upgraded master recently. The master was upgraded without any problem, but the slave was crashed and could not be started. our upgrade steps: 1. backup existing MySQL databases (database backup) and 5.5.11 server (do not delete 5.5.11) on all master and slaves 2. install 5.5.25a server on Slave server 3. stop the Slave 4. start the slave from 5.5.25a 5. run mysql_upgrade utility against Slave 6. restart Slave from 5.5.25a and check for any issue from *.err logfile 7. repeat above steps 2 to 6 for all Slaves ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 8. install 5.5.25a server on Master server 9. stop the Master 10.start the Master from 5.5.25a 11.run mysql_upgrade utility against Master 12.restart Master from 5.5.25a and check for any issue from *.err logfile 13.Check replication status How to repeat: It caused major problem and the servers had to be shutdown for almost a day to rebuild the slave from master. Not dare to repeat! Suggested fix: MySQL must provide a clear upgrade instructions to make it more of usable product of easier upgrading.