Bug #38722 mysqld showing connected when it is not
Submitted: 11 Aug 2008 15:22 Modified: 13 Aug 2008 4:07
Reporter: Lig Isler-Turmelle Email Updates:
Status: Closed Impact on me:
None 
Category:MySQL Enterprise Monitor: Server Severity:S2 (Serious)
Version:2.0.0.7017 OS:Mac OS X (10.5.4)
Assigned to: Darren Oldag CPU Architecture:Any
Triage: D1 (Critical)

[11 Aug 2008 15:22] Lig Isler-Turmelle
Description:
server and agent status have green lights when the server they are monitoring is not started never mind connected.

Note:  Had the servers up and being monitored earlier (log showed that the heartbeat was working with the servers).  I have 2 servers (5.0 and 5.1) running concurrently on my laptop.  

command line:
ligaya-turmelles-macbook-pro:~ ligaya$ ps aux |grep mysql
ligaya     356   0.0  0.0    75492    440 s000  R+   10:44AM   0:00.00 grep mysql
ligaya-turmelles-macbook-pro:~ ligaya$ /Applications/mysql/MEM2/agent/etc/init.d/mysql-monitor-agent start
Starting MySQL Enterprise agent service...
 SUCCESS! 
ligaya-turmelles-macbook-pro:~ ligaya$ ps aux |grep mysql
ligaya     378   0.0  0.0    75492    412 s000  R+   10:44AM   0:00.00 grep mysql
ligaya     376   0.0  0.1    84736   2036   ??  S    10:44AM   0:00.01 /Applications/mysql/MEM2/agent/sbin/mysql-monitor-agent --defaults-file=/Applications/mysql/MEM2/agent/etc/mysql-monitor-agent.ini --daemon --pid-file=/Applications/mysql/MEM2/agent/mysql-monitor-agent.pid
ligaya-turmelles-macbook-pro:~ ligaya$ /Applications/mysql/MEM2/monitor/mysqlmonitorctl.sh start
/Applications/mysql/MEM2/monitor/mysqlmonitorctl.sh : mysql  started
chown: /Applications/mysql/MEM2/monitor/mysql/data/mysqld.log: Operation not permitted
Starting mysqld daemon with databases from /Applications/mysql/MEM2/monitor/mysql/data/
Using CATALINA_BASE:   /Applications/mysql/MEM2/monitor/apache-tomcat
Using CATALINA_HOME:   /Applications/mysql/MEM2/monitor/apache-tomcat
Using CATALINA_TMPDIR: /Applications/mysql/MEM2/monitor/apache-tomcat/temp
Using JRE_HOME:       /System/Library/Frameworks/JavaVM.framework/Versions/1.5.0/Home
ligaya-turmelles-macbook-pro:~ ligaya$ ps aux |grep mysql
ligaya     427  11.3  3.9  1167372  81088 s000  S    10:45AM   0:07.49 /System/Library/Frameworks/JavaVM.framework/Versions/1.5.0/Home/bin/java -Xmx512m -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Djava.util.logging.config.file=/Applications/mysql/MEM2/monitor/apache-tomcat/conf/logging.properties -Djava.endorsed.dirs=/Applications/mysql/MEM2/monitor/apache-tomcat/endorsed -classpath :/Applications/mysql/MEM2/monitor/apache-tomcat/bin/bootstrap.jar:/Applications/mysql/MEM2/monitor/apache-tomcat/bin/commons-logging-api.jar -Dcatalina.base=/Applications/mysql/MEM2/monitor/apache-tomcat -Dcatalina.home=/Applications/mysql/MEM2/monitor/apache-tomcat -Djava.io.tmpdir=/Applications/mysql/MEM2/monitor/apache-tomcat/temp org.apache.catalina.startup.Bootstrap start
ligaya     414   0.2  1.5   226924  30768 s000  S    10:45AM   0:00.69 /Applications/mysql/MEM2/monitor/mysql/bin/mysqld --defaults-file=/Applications/mysql/MEM2/monitor/mysql/my.cnf --basedir=/Applications/mysql/MEM2/monitor/mysql/ --datadir=/Applications/mysql/MEM2/monitor/mysql/data/ --user=mysql --pid-file=/Applications/mysql/MEM2/monitor/mysql/data/mysqld.pid --port=13306 --socket=/Applications/mysql/MEM2/monitor/mysql/tmp/mysql.sock
ligaya     376   0.1  0.1    84760   2152   ??  S    10:44AM   0:00.04 /Applications/mysql/MEM2/agent/sbin/mysql-monitor-agent --defaults-file=/Applications/mysql/MEM2/agent/etc/mysql-monitor-agent.ini --daemon --pid-file=/Applications/mysql/MEM2/agent/mysql-monitor-agent.pid
ligaya     429   0.0  0.0    75492    464 s000  R+   10:45AM   0:00.00 grep mysql
ligaya     381   0.0  0.0    75964    792 s000  S    10:45AM   0:00.02 /bin/sh /Applications/mysql/MEM2/monitor/mysql/bin/mysqld_safe --defaults-file=/Applications/mysql/MEM2/monitor/mysql/my.cnf --log-error=/Applications/mysql/MEM2/monitor/mysql/data/mysqld.log --user=mysql

Image of what the dashboard looks like immediately after the above commands:
http://khankennels.com/stuff/picture_1.png

Firefox: 2.0.0.16
about:config - http://khankennels.com/stuff/picture_2.png

How to repeat:
1) have the monitor up and running connected to the servers with all going well.
2) shut down the monitor and the agents
3) shut down the servers
4) bring back up the agents
5) bring back up the monitor
6) view the dashboard - all green lights
[11 Aug 2008 18:03] Diego Medina
Verified as described
[12 Aug 2008 0:52] Darren Oldag
we depend on the server.reachable data value reported from the agent in order to know,
well, if the server is reachable or not.  in some cases, the last value we have stored
for it cannot be trusted.

1) in case the transport (agent) goes down or stops monitoring the mysqld, we have to
   assume the reachability is unknown.
2) in case the service manager restarts, we should also assume the reachability is 
   unknown, until an agent actually checks in for a server.

for #1, we watch for an agent.reachable to go to 'shutdown' or 'timedout', in which
case we mark any servers it reported on as server.reachable = unknown.

for #2, at service manager startup, we mark all servers as 'unknown' because until
an agent pings in for the server, we really don't want to assume if it is reachable
or not.
[13 Aug 2008 4:07] Diego Medina
Fixed on 2.0.0.7024