Description:
The glibc malloc library is known to have issues with threading.
How to repeat:
From http://ozlabs.org/~anton/linux/sysbench/
After looking at the gdb backtraces again, it was noticed that the code in heap/* was doing memory allocation while under the THR_LOCK_heap mutex (see http://bugs.mysql.com/27062). In fact the glibc free routine was blocking on its own mutex when holding the THR_LOCK_heap mutex:
#0 0x0fc538a8 in __lll_lock_wait () from /lib/tls/libc.so.6
#1 0x0fbdb0a0 in free () from /lib/tls/libc.so.6
#2 0x1042b458 in my_no_flags_free ()
#3 0x102522c4 in ha_heap::create ()
#4 0x102531c0 in ha_heap::open ()
#5 0x1024db80 in handler::ha_open ()
#6 0x101c9ae0 in create_tmp_table ()
#7 0x101dc6b0 in JOIN::optimize ()
#8 0x101e1ccc in mysql_select ()
#9 0x101e26c8 in handle_select ()
#10 0x1018bb6c in mysql_execute_command ()
#11 0x101ec414 in Prepared_statement::execute ()
#12 0x101ec8c4 in mysql_stmt_execute ()
#13 0x10192d5c in dispatch_command ()
#14 0x10193f94 in do_command ()
#15 0x101949c0 in handle_one_connection ()
#16 0x0ff50618 in start_thread () from /lib/tls/libpthread.so.0
The glibc malloc library is known to have issues with threading. Since these allocation calls are all wrapped in the THR_LOCK_heap mutex, these negative effects could be multiplied.
Google have written an alternate malloc library that is known to perform better with threads. Debian has packaged it, so it is a simple matter of downloading and using LD_PRELOAD again:
# apt-get install libgoogle-perftools0
# LD_PRELOAD=/usr/lib/libtcmalloc.so:override_sched_setscheduler.so /usr/sbin/mysqld
Replacing the glibc malloc with the google malloc has fixed the scalability issue.
Suggested fix:
Review and confirm proper behaviour, then recommend using tcmalloc instead of default glibc malloc library.
