Bug #27063 glibc malloc threading problems, Google's tcmalloc appears to work well
Submitted: 13 Mar 2007 3:25 Modified: 13 Mar 2007 7:31
Reporter: Arjen Lentz Email Updates:
Status: Duplicate Impact on me:
None 
Category:MySQL Server: Compiling Severity:S5 (Performance)
Version:any OS:Any (any)
Assigned to: CPU Architecture:Any
Tags: glibc, malloc, scaling, tcmalloc, threading

[13 Mar 2007 3:25] Arjen Lentz
Description:
The glibc malloc library is known to have issues with threading.

How to repeat:
From http://ozlabs.org/~anton/linux/sysbench/

After looking at the gdb backtraces again, it was noticed that the code in heap/* was doing memory allocation while under the THR_LOCK_heap mutex (see http://bugs.mysql.com/27062). In fact the glibc free routine was blocking on its own mutex when holding the THR_LOCK_heap mutex:

#0  0x0fc538a8 in __lll_lock_wait () from /lib/tls/libc.so.6
#1  0x0fbdb0a0 in free () from /lib/tls/libc.so.6
#2  0x1042b458 in my_no_flags_free ()
#3  0x102522c4 in ha_heap::create ()
#4  0x102531c0 in ha_heap::open ()
#5  0x1024db80 in handler::ha_open ()
#6  0x101c9ae0 in create_tmp_table ()
#7  0x101dc6b0 in JOIN::optimize ()
#8  0x101e1ccc in mysql_select ()
#9  0x101e26c8 in handle_select ()
#10 0x1018bb6c in mysql_execute_command ()
#11 0x101ec414 in Prepared_statement::execute ()
#12 0x101ec8c4 in mysql_stmt_execute ()
#13 0x10192d5c in dispatch_command ()
#14 0x10193f94 in do_command ()
#15 0x101949c0 in handle_one_connection ()
#16 0x0ff50618 in start_thread () from /lib/tls/libpthread.so.0

The glibc malloc library is known to have issues with threading. Since these allocation calls are all wrapped in the THR_LOCK_heap mutex, these negative effects could be multiplied.

Google have written an alternate malloc library that is known to perform better with threads. Debian has packaged it, so it is a simple matter of downloading and using LD_PRELOAD again:

# apt-get install libgoogle-perftools0
# LD_PRELOAD=/usr/lib/libtcmalloc.so:override_sched_setscheduler.so /usr/sbin/mysqld

Replacing the glibc malloc with the google malloc has fixed the scalability issue.

Suggested fix:
Review and confirm proper behaviour, then recommend using tcmalloc instead of default glibc malloc library.
[13 Mar 2007 7:31] Sergei Golubchik
this is a duplicate of bug#25244