Description:
Setup: Write-only master, 3 slaves replicating from it, master and one slave have 6G RAM, slave2 has 3G, slave3 has 3.5G.
After upgrading to 5.1.23 we have had several crashes, and looking at merlin RAM graphs we see a slow drop in free memory until the server crashes. This crash is usually reported as a failure to allocate memory, however at the time the crash occurs merlin graphs (and top) show anything up to 1G RAM still free.
This was first noticed when investigating a crash originally thought to be related to proxy load balancer, although the current crashes have occurred without heavy proxy load.
Most recent crash (on slave) reports the following stack trace:
0x81eeff7 handle_segfault + 535
0x8418402 ut_malloc_low + 258
0x84184a1 ut_malloc + 33
0x83d0fee os_aio_simulated_handle + 542
0x83a15fc fil_aio_wait + 76
0x8400298 io_handler_thread + 24
0x2ef45b (?)
0x24723e (?)
Additional data points:
When full load was routed through proxy load balancer, the memory leak appeared to worsen, although this may have been coincidental. The exact same queries were being processed, the only difference being the load balancer being placed between application and slaves.
Merlin only reports 3.5G total memory even on those servers with 6G RAM.
Points at which memory curve inflects appear to coincide with operations on temporary tables.
How to repeat:
Install 5.1.23 with heavy load on multiple replicated slaves, monitor RAM usage.
