MySQL Bugs: #56092: Agents get many timeouts from the service manager on large deployment

Bug #56092	Agents get many timeouts from the service manager on large deployment
Submitted:	18 Aug 2010 17:49	Modified:	9 Jan 2015 15:56
Reporter:	Diego Medina	Email Updates:
Status:	No Feedback	Impact on me:	None
Category:	MySQL Enterprise Monitor: Server	Severity:	S2 (Serious)
Version:	2.2.2.1729	OS:	Any
Assigned to:	Assigned Account	CPU Architecture:	Any

Description:
I am monitoring 203 servers, 200 of those agents are on a server with 96 cores and about 12 partitions. all 200 agents have a different host-id.

all my agents are getting timeouts, which causes me to miss some query analyzer data and I also see gaps of hours in the graphs.

2 of those agents are running heavy quan (repo and sugarcrm)

How to repeat:
1- run my large deployment

Diego Medina writes: 
this is to keep track of the work Mark and Oldag were/are doing

Diego Medina writes: 
dump_1.txt is a thread dump when all agents were using ssl (they no longer use ssl)



Attachment: 10431_dump_1.txt (text/plain), 903.08 KiB.

Diego Medina writes: 
Heap dump not using ssl

https://intranet.mysql.com/~dmedina/tmp/threaddump.24710.10.out.bz2

(I did not attach this file because it was too big for Jira)

Diego Medina writes: 
Show engine innodb status output



Attachment: 10432_innodb.171.log (text/plain), 35.88 KiB.

Diego Medina writes: 
threaddump while getting the innodb status output

Attachment: 10433_threaddump.24710.10.out.bz2 (application/x-bzip2, text), 20.28 KiB.

Diego Medina writes: 
For all the other files visit:

https://intranet.mysql.com/~dmedina/tmp/threaddump.24710.1.out.bz2 to

https://intranet.mysql.com/~dmedina/tmp/threaddump.24710.100.out.bz2

and

from

https://intranet.mysql.com/~dmedina/tmp/innodb.1.log

to 

https://intranet.mysql.com/~dmedina/tmp/innodb.200.log