Bug #35771 MySQL Enterprise Monitor does not update replication settings correctly
Submitted: 2 Apr 2008 13:56 Modified: 8 Aug 2008 14:49
Reporter: Bogdan Kecman Email Updates:
Status: Closed Impact on me:
Category:MySQL Enterprise Monitor: Advisors/Rules Severity:S3 (Non-critical)
Version:1.3 OS:Any
Assigned to: Andy Bang CPU Architecture:Any

[2 Apr 2008 13:56] Bogdan Kecman
After slave becomes the master adviser still check the slave status of the server (that is now master) hence reports wrong error

How to repeat:
Set up replication

db1 - master
db2 - slave

Set up MEM to monitor it

Remove db1 from the system, add db3 set config now as

db2 - master
db3 - slave

now the "adviser" send email about "slave not running on db2" 
and db2 is a master - not slave, it is normal that "slave is not running"

Note that dashboard correctly show that db2 is master and db3 is slave and that db1 is out of the picture

Suggested fix:
[7 Apr 2008 16:02] Mark Leith
A question about this:

Was a RESET SLAVE / CHANGE MASTER TO MASTER_HOST = ''; performed on the new master so that all slave settings were totally removed?

If so, then this is likely the rule that checks the Slave_running SHOW GLOBAL STATUS variable. If the server is no longer a slave then in this case the 'workaround' (or suggested solution?) is to simply disable the advisor for any system which is not a slave. 

There may be some more we could do for this however, so I will look in to this further.
[8 Apr 2008 14:05] Simon Mudd
I've checked with my colleague and he tells me just a CHANGE MASTER TO

[20 May 2008 23:05] Andy Bang
The problem is that the "Slave Has Been Stopped" rule depends on the Slave_running status variable reported by SHOW GLOBAL STATUS, but that variable has no idea whether the underlying server is configured as a slave or not.  Ideally, this rule should depend on a variable available via SHOW SLAVE STATUS so it could not be scheduled against anything but a slave, and would stop working when a server is no longer configured as a slave (i.e. doesn't return any results from SHOW SLAVE STATUS).  Since we already have other and probably better rules for this (e.g. "Slave I/O Thread Not Running" and "Slave SQL Thread Not Running"), the best thing to do is just remove this rule altogether.

Unfortunately, with things as they are in the 1.x releases of the Enterprise Monitor, people who upgrade to a new rule bundle that doesn't have this rule will still have it because we just update rules, we don't actually remove them, so it's still there from the import of an older bundle.  So Support will just have to advise customers not to schedule that rule, or to unschedule it if it's already scheduled, and we need to update the documentation accordingly.
[20 May 2008 23:14] Andy Bang
Committed revision 9115.
[21 May 2008 6:40] Simon Mudd
Thanks for the extra comments. If MEM doesn't currently support removing certain rules then I guess the answer is to adjust it (later) to be able to delete/remove/disable them permanently.

Expecting the client to do manual changes (disabling obsolete rules) is not a good idea and just won't work very well. This type of problem will almost certainly happen with other sorts of rules in the future so having a generic method to deal with it would save you time and effort.
[21 May 2008 21:56] Andy Bang
Absolutely, it's on the list.
[29 May 2008 20:53] Andy Bang
When we remove the rule from the bundle, we also remove its resource text.  This is fine if you don't have the rule from a previous import of an older bundle, but if you do have it because of that import you end up with the rule still there but no text to show on screen, so you just end up seeing the resources keys stored with the rule rather than the resource values that are meant for humans.  So I put the rule back in the bundle, but added a reference to a slave-only data collection item so you cannot schedule this rule against anything that isn't a slave (i.e. anything that doesn't return results from SHOW SLAVE STATUS).

Committed revision 9134.
[4 Jun 2008 15:26] Marcos Palacios
Verified fixed in advisor build
[23 Jul 2008 11:07] Tony Bedford
Set to "verified" and "target" of 2.0 as requested.

Also added a entry to the 1.3 changelog:

MySQL Enterprise Monitor did not update replication settings correctly. After a slave became the master, the Adviser still referred to it as a slave.
[29 Jul 2008 23:46] Andy Bang
In 2.0: Pushed up to revision 216.
[6 Aug 2008 20:24] Marcos Palacios
Verified fixed in advisor build
[8 Aug 2008 14:49] Tony Bedford
Updated 2.0 changelog accordingly.
[13 Jun 2010 20:54] Roel Van de Paar
Also see bug #50889