MySQL Bugs: #35771: MySQL Enterprise Monitor does not update replication settings correctly

Bug #35771	MySQL Enterprise Monitor does not update replication settings correctly
Submitted:	2 Apr 2008 13:56	Modified:	8 Aug 2008 14:49
Reporter:	Bogdan Kecman	Email Updates:
Status:	Closed	Impact on me:	None
Category:	MySQL Enterprise Monitor: Advisors/Rules	Severity:	S3 (Non-critical)
Version:	1.3	OS:	Any
Assigned to:	Andy Bang	CPU Architecture:	Any

Description:
After slave becomes the master adviser still check the slave status of the server (that is now master) hence reports wrong error

How to repeat:
Set up replication

db1 - master
db2 - slave

Set up MEM to monitor it

Remove db1 from the system, add db3 set config now as

db2 - master
db3 - slave

now the "adviser" send email about "slave not running on db2" 
and db2 is a master - not slave, it is normal that "slave is not running"

Note that dashboard correctly show that db2 is master and db3 is slave and that db1 is out of the picture

Suggested fix:
n/a

A question about this:

Was a RESET SLAVE / CHANGE MASTER TO MASTER_HOST = ''; performed on the new master so that all slave settings were totally removed?

If so, then this is likely the rule that checks the Slave_running SHOW GLOBAL STATUS variable. If the server is no longer a slave then in this case the 'workaround' (or suggested solution?) is to simply disable the advisor for any system which is not a slave. 

There may be some more we could do for this however, so I will look in to this further.

I've checked with my colleague and he tells me just a CHANGE MASTER TO
''.

Simon

The problem is that the "Slave Has Been Stopped" rule depends on the Slave_running status variable reported by SHOW GLOBAL STATUS, but that variable has no idea whether the underlying server is configured as a slave or not.  Ideally, this rule should depend on a variable available via SHOW SLAVE STATUS so it could not be scheduled against anything but a slave, and would stop working when a server is no longer configured as a slave (i.e. doesn't return any results from SHOW SLAVE STATUS).  Since we already have other and probably better rules for this (e.g. "Slave I/O Thread Not Running" and "Slave SQL Thread Not Running"), the best thing to do is just remove this rule altogether.

Unfortunately, with things as they are in the 1.x releases of the Enterprise Monitor, people who upgrade to a new rule bundle that doesn't have this rule will still have it because we just update rules, we don't actually remove them, so it's still there from the import of an older bundle.  So Support will just have to advise customers not to schedule that rule, or to unschedule it if it's already scheduled, and we need to update the documentation accordingly.

Committed revision 9115.

Thanks for the extra comments. If MEM doesn't currently support removing certain rules then I guess the answer is to adjust it (later) to be able to delete/remove/disable them permanently.

Expecting the client to do manual changes (disabling obsolete rules) is not a good idea and just won't work very well. This type of problem will almost certainly happen with other sorts of rules in the future so having a generic method to deal with it would save you time and effort.

Absolutely, it's on the list.

When we remove the rule from the bundle, we also remove its resource text.  This is fine if you don't have the rule from a previous import of an older bundle, but if you do have it because of that import you end up with the rule still there but no text to show on screen, so you just end up seeing the resources keys stored with the rule rather than the resource values that are meant for humans.  So I put the rule back in the bundle, but added a reference to a slave-only data collection item so you cannot schedule this rule against anything that isn't a slave (i.e. anything that doesn't return results from SHOW SLAVE STATUS).

Committed revision 9134.

Verified fixed in advisor build 1.3.0.9137.

Set to "verified" and "target" of 2.0 as requested.

Also added a entry to the 1.3 changelog:

MySQL Enterprise Monitor did not update replication settings correctly. After a slave became the master, the Adviser still referred to it as a slave.

In 2.0: Pushed up to revision 216.

Verified fixed in advisor build 2.0.0.7020.

Updated 2.0 changelog accordingly.

Also see bug #50889