MySQL Bugs: #17292: MEMORY tables do not release memory to OS when dropped on Solaris 9

Bug #17292	MEMORY tables do not release memory to OS when dropped on Solaris 9
Submitted:	10 Feb 2006 1:32	Modified:	21 Mar 2006 23:19
Reporter:	Lachlan Mulcahy	Email Updates:
Status:	Not a Bug	Impact on me:	None
Category:	MySQL Server	Severity:	S2 (Serious)
Version:	4.1.18	OS:	Solaris (Solaris 9)
Assigned to:	Ingo Strüwing	CPU Architecture:	Any

Description:
If a table is created using the MEMORY storage engine, when the table or database the table belongs to is dropped the memory consumed by the MEMORY table is not freed back to the Operating System. This only seems to occur on Solaris 9, Linux frees the memory correctly.

How to repeat:
Create a MEMORY table on MySQL 4.1.18 on Solaris 9
Observe mysqld processes memory consumption
Populate the table until it is sizeable.
Observe mysqld processes memory consumption again
DROP the table
Observe mysqld processes memory consumption again -- it does not return to the original value, or even close to it.

You can repeat the above steps on Linux and observe the memory being correctly freed.

Suggested fix:
unknown

I did not spend any time on investigating this yet. But from my immediate believe how MEMORY tables work, I would say it's not a bug.

MEMORY tables have been called HEAP tables before. This term was technically more precise as we allocate the memory from the "heap". The classical way to segment the processes address space is text, heap and stack. Text (the executable code) and heap (data) start from low addresses, stack starts from high addresses. Heap grows up and stack grows down. Regardless if the program uses more heap or more stack, the program can utilize the maximum amount of address space.

MEMORY tables allocate their address space from the heap by calling malloc() and return it with free(). These functions are libc functions and the way they work depends on the OSs implementation. So it is operating system dependend if free() will return the address space to the operating system. And even if it implements this, it can return the space only if the freed segment is the highest segment in the heap (The heap address space is never segmented in UNIX like OSs). That is no other malloced data with higher addresses do exist.

When I find time to deal with this bug, my first task will be to write a small program that simply mallocs some space and returns it. I will run it on Solaris and Linux and see if Solaris can release it at all. If not, I will reject it as "not a bug". If yes, I will have more effort to see if other malloced segments exits in the Solaris case and why.

Regards
Ingo

See explanation at: http://forum.sun.com/thread.jspa?threadID=11027&messageID=31172

This was confirmed officially by Sun.

Found simplified, additional forum entry.  This appears to be a 'solaris' (v9 and v10) specific issue.

   http://forums.sun.com/thread.jspa?forumID=840&threadID=5090911

Effectivly, unless the developer changes from using malloc() and free() to use mmap() and munmap() the allocated memory remains 'assigned' to the process.  
This is good for processes that malloc/free often and are short-lived.  However if you have a Database that is on-going to execute for weeks or months on end.  This hurts.  It hurts worse when the database server has to competes against other [short-lived] memory intensive processes on the server.  To add further misery, if there is even 1 SQL query (ie. accidential cartesian) that requires the DB to request lots of RAM, increasing the DB footprint, you've got yourself effectively a 'memory' leak until you bounce the database.  Those other short-lived process could be forced to exist only in swap and potentially finish exhausting all RAM.