Description:
While testing the Query Analyzer on the production databases on FC6 backing a very busy website, the agent RAM usage saw almost linear memory growth until no new queries were sent to the proxy port. At that point, RAM usage growth stopped, but no substantial amount of memory was returned to the OS until the agent/proxy process terminated.
Some increased memory usage is expected behavior when using the query analyzer, especially since we need a working set of data to produce data; but this particular machine has query analyzer disabled now. The moment it was disabled, memory increase stopped, but even hours later, no memory was given back to the system.
Apparently the standard allocator (glibc's malloc, Linux 2.6.22, Fedora Core 6) does not or cannot do this. In fact, not freeing RAM is something we can now reproduce on several platforms. In trying to reproduce this load in our lab, we are unable to see unbounded growth; however, on some systems we do see more RAM consumption than we'd expect.
Further instrumentation has shown that libxml is doing a tremendous amount of malloc()ing of small bits of memory, far and away exceeding the number of allocations that other components in the agent (curl, glib, etc) are performing. Therefore we believe that the memory growth is actually heap fragmentation and are investigating how to avoid it.
How to repeat:
(see description)
Suggested fix:
Investigating:
- replacing libxml
- looking at different allocators
