Bug #54932 wrong "Excessive Memory Usage" alerts on AIX
Submitted: 1 Jul 2010 11:48 Modified: 9 Jan 2015 14:45
Reporter: Andrii Nikitin Email Updates:
Status: Won't fix Impact on me:
Category:MySQL Enterprise Monitor: Agent Severity:S3 (Non-critical)
Version:2.2.3 OS:IBM AIX
Assigned to: Assigned Account CPU Architecture:Any

[1 Jul 2010 11:48] Andrii Nikitin
AIX always tries to utilize all unused RAM for file cache, so it always looks that "free RAM" is very small. 

But when some application requests memory, the file cache will be reduced, so application easily receives claimed size.

In particular this behavior is documented in 'man vmstat':
            Size of the free list. Note: A large portion of real memory is utilized as a cache for file system data. It is not unusual for the size of the free list to remain small.

It may not obvious how to report Free RAM graph, but certainly alert "RAM Usage Excessive" shouldn't be shown if most of RAM is used for File cache.

Example output:
support@sup-aix53:~$ vmstat -v
               487424 memory pages
               461625 lruable pages
                20235 free pages
                    1 memory pools
               126349 pinned pages
                 80.0 maxpin percentage
                 20.0 minperm percentage
                 80.0 maxperm percentage
                 62.3 numperm percentage
               287812 file pages
                  0.0 compressed percentage
                    0 compressed pages
                 62.3 numclient percentage
                 80.0 maxclient percentage
               287812 client pages
                    0 remote pageouts scheduled
                 4275 pending disk I/Os blocked with no pbuf
                    0 paging space I/Os blocked with no psbuf
                 2228 filesystem I/Os blocked with no fsbuf
                 1591 client filesystem I/Os blocked with no fsbuf
                16264 external pager filesystem I/Os blocked with no fsbuf
                    0 Virtualized Partition Memory Page Faults
                 0.00 Time resolving virtualized partition memory page faults

Numbers represent 4K pages:

1.9G RAM total (487424 memory pages)
80M RAM free   (20235 free pages)
1.15G used for file cache (287812 file pages)

So MEM thinks that only 80M RAM is available and raises critical alert "RAM Usage Excessive". 
But in fact there is still more than 1G which processes can easily allocate.

How to repeat:
Start agent on busy AIX system - "RAM Usage Excessive" alert will be generated.

Suggested fix:
Include file cache into calculation. (E.g. collect additional item "file pages" from "vmstat -v" output).