Bug #53794 Weird printout from test suite
Submitted: 19 May 2010 10:43 Modified: 24 May 2010 16:57
Reporter: Ingo Strüwing Email Updates:
Status: Duplicate Impact on me:
None 
Category:MySQL Server: General Severity:S3 (Non-critical)
Version:5.1, 5.1-bugteam, mysql-pe OS:Linux (Ubuntu 9.10 64bit)
Assigned to: Assigned Account CPU Architecture:Any

[19 May 2010 10:43] Ingo Strüwing
Description:
When I run the test suite on local branches of mysql-5.1, mysql-5.1-bugteam, mysql-pe, I see printouts from glibc on the controlling terminal (this does not go to stdout, neither strderr). This is Ubuntu 9.10, 64 bit with a -debug-max build. It repeats a couple of times in a test suite run, but probably not every time mysqld stops. After some experiments, I found that main.flush_read_lock_kill is one candidate to provoke this.

main.flush_read_lock_kill                w2 [ pass ]    254
*** glibc detected *** /home2/mydev/bzrroot/mysql-5.1-main/sql/mysqld: double free or corruption (fasttop): 0x000000000310f8c0 ***
======= Backtrace: =========
/lib/libc.so.6[0x7f2ff2c2add6]
/lib/libc.so.6(cfree+0x6c)[0x7f2ff2c2f74c]
/home2/mydev/bzrroot/mysql-5.1-main/sql/mysqld[0xb8133c]
/home2/mydev/bzrroot/mysql-5.1-main/sql/mysqld[0xb80da8]
/home2/mydev/bzrroot/mysql-5.1-main/sql/mysqld(_db_end_+0x15d)[0xb81063]
/home2/mydev/bzrroot/mysql-5.1-main/sql/mysqld(my_end+0x36e)[0xb4bcaf]
/home2/mydev/bzrroot/mysql-5.1-main/sql/mysqld(main+0x63c)[0x6c95a7]
/lib/libc.so.6(__libc_start_main+0xfd)[0x7f2ff2bd3abd]
/home2/mydev/bzrroot/mysql-5.1-main/sql/mysqld[0x5c4e69]
======= Memory map: ========
00400000-00e87000 r-xp 00000000 08:19 7192802                            /home2/mydev/bzrroot/mysql-5.1-main/sql/mysqld
01087000-0109c000 r--p 00a87000 08:19 7192802                            /home2/mydev/bzrroot/mysql-5.1-main/sql/mysqld
0109c000-011ce000 rw-p 00a9c000 08:19 7192802                            /home2/mydev/bzrroot/mysql-5.1-main/sql/mysqld
011ce000-011ed000 rw-p 00000000 00:00 0 
03108000-03208000 rw-p 00000000 00:00 0                                  [heap]
7f2fec000000-7f2fec021000 rw-p 00000000 00:00 0 
7f2fec021000-7f2ff0000000 ---p 00000000 00:00 0 
7f2ff215c000-7f2ff215d000 ---p 00000000 00:00 0 
7f2ff215d000-7f2ff219d000 rw-p 00000000 00:00 0 
7f2ff219d000-7f2ff21b3000 r-xp 00000000 08:01 89073                      /lib/libgcc_s.so.1
7f2ff21b3000-7f2ff23b2000 ---p 00016000 08:01 89073                      /lib/libgcc_s.so.1
7f2ff23b2000-7f2ff23b3000 r--p 00015000 08:01 89073                      /lib/libgcc_s.so.1
7f2ff23b3000-7f2ff23b4000 rw-p 00016000 08:01 89073                      /lib/libgcc_s.so.1
7f2ff23b4000-7f2ff23b5000 ---p 00000000 00:00 0 
7f2ff23b5000-7f2ff2bb5000 rw-p 00000000 00:00 0 
7f2ff2bb5000-7f2ff2d1b000 r-xp 00000000 08:01 89865                      /lib/libc-2.10.1.so
7f2ff2d1b000-7f2ff2f1a000 ---p 00166000 08:01 89865                      /lib/libc-2.10.1.so
7f2ff2f1a000-7f2ff2f1e000 r--p 00165000 08:01 89865                      /lib/libc-2.10.1.so
7f2ff2f1e000-7f2ff2f1f000 rw-p 00169000 08:01 89865                      /lib/libc-2.10.1.so
7f2ff2f1f000-7f2ff2f24000 rw-p 00000000 00:00 0 
7f2ff2f24000-7f2ff2fa6000 r-xp 00000000 08:01 89880                      /lib/libm-2.10.1.so
7f2ff2fa6000-7f2ff31a6000 ---p 00082000 08:01 89880                      /lib/libm-2.10.1.so
7f2ff31a6000-7f2ff31a7000 r--p 00082000 08:01 89880                      /lib/libm-2.10.1.so
7f2ff31a7000-7f2ff31a8000 rw-p 00083000 08:01 89880                      /lib/libm-2.10.1.so
7f2ff31a8000-7f2ff31be000 r-xp 00000000 08:01 89887                      /lib/libnsl-2.10.1.so
7f2ff31be000-7f2ff33be000 ---p 00016000 08:01 89887                      /lib/libnsl-2.10.1.so
7f2ff33be000-7f2ff33bf000 r--p 00016000 08:01 89887                      /lib/libnsl-2.10.1.so
7f2ff33bf000-7f2ff33c0000 rw-p 00017000 08:01 89887                      /lib/libnsl-2.10.1.so
7f2ff33c0000-7f2ff33c2000 rw-p 00000000 00:00 0 
7f2ff33c2000-7f2ff33cb000 r-xp 00000000 08:01 89878                      /lib/libcrypt-2.10.1.so
7f2ff33cb000-7f2ff35cb000 ---p 00009000 08:01 89878                      /lib/libcrypt-2.10.1.so
7f2ff35cb000-7f2ff35cc000 r--p 00009000 08:01 89878                      /lib/libcrypt-2.10.1.so
7f2ff35cc000-7f2ff35cd000 rw-p 0000a000 08:01 89878                      /lib/libcrypt-2.10.1.so
7f2ff35cd000-7f2ff35fb000 rw-p 00000000 00:00 0 
7f2ff35fb000-7f2ff35fd000 r-xp 00000000 08:01 89879                      /lib/libdl-2.10.1.so
7f2ff35fd000-7f2ff37fd000 ---p 00002000 08:01 89879                      /lib/libdl-2.10.1.so
7f2ff37fd000-7f2ff37fe000 r--p 00002000 08:01 89879                      /lib/libdl-2.10.1.so
7f2ff37fe000-7f2ff37ff000 rw-p 00003000 08:01 89879                      /lib/libdl-2.10.1.so
7f2ff37ff000-7f2ff3815000 r-xp 00000000 08:01 89118                      /lib/libz.so.1.2.3.3
7f2ff3815000-7f2ff3a14000 ---p 00016000 08:01 89118                      /lib/libz.so.1.2.3.3
7f2ff3a14000-7f2ff3a15000 r--p 00015000 08:01 89118                      /lib/libz.so.1.2.3.3
7f2ff3a15000-7f2ff3a16000 rw-p 00016000 08:01 89118                      /lib/libz.so.1.2.3.3
7f2ff3a16000-7f2ff3a2d000 r-xp 00000000 08:01 89895                      /lib/libpthread-2.10.1.so
7f2ff3a2d000-7f2ff3c2c000 ---p 00017000 08:01 89895                      /lib/libpthread-2.10.1.so
7f2ff3c2c000-7f2ff3c2d000 r--p 00016000 08:01 89895                      /lib/libpthread-2.10.1.so
7f2ff3c2d000-7f2ff3c2e000 rw-p 00017000 08:01 89895                      /lib/libpthread-2.10.1.so
7f2ff3c2e000-7f2ff3c32000 rw-p 00000000 00:00 0 
7f2ff3c32000-7f2ff3c51000 r-xp 00000000 08:01 89099                      /lib/ld-2.10.1.so
7f2ff3c66000-7f2ff3c67000 ---p 00000000 00:00 0 
7f2ff3c67000-7f2ff3ca7000 rw-p 00000000 00:00 0 
7f2ff3ca7000-7f2ff3ca8000 ---p 00000000 00:00 0 
7f2ff3ca8000-7f2ff3ce8000 rw-p 00000000 00:00 0 
7f2ff3ce8000-7f2ff3ce9000 ---p 00000000 00:00 0 
7f2ff3ce9000-7f2ff3d29000 rw-p 00000000 00:00 0 
7f2ff3e2a000-7f2ff3e2e000 rw-p 00000000 00:00 0 
7f2ff3e4d000-7f2ff3e50000 rw-p 00000000 00:00 0 
7f2ff3e50000-7f2ff3e51000 r--p 0001e000 08:01 89099                      /lib/ld-2.10.1.so
7f2ff3e51000-7f2ff3e52000 rw-p 0001f000 08:01 89099                      /lib/ld-2.10.1.so
7fffee24b000-7fffee261000 rw-p 00000000 00:00 0                          [stack]
7fffee2ec000-7fffee2ed000 r-xp 00000000 00:00 0                          [vdso]
ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0                  [vsyscall]
------------------------------------------------------------
The servers were restarted 0 times

How to repeat:
Note:
      #define MYSQL_SERVER_VERSION            "5.1.48"
      AC_INIT([MySQL Server], [5.1.48], [], [mysql])

      OS: GNU/Linux/x86_64
      OS: kernel 2.6.31-21-generic  SMP
      gcc (Ubuntu 4.4.1-4ubuntu9) 4.4.1
      libc6 

bzr branch bzr+ssh://bk-internal.mysql.com/bzrroot/server/mysql-5.1 mysql-5.1-main
cd mysql-5.1-main
BUILD/compile-pentium64-debug-max --with-debug=full
cd mysql-test
./mysql-test-run.pl main.flush_read_lock_kill
[19 May 2010 10:47] Ingo Strüwing
Oops, I meant to add "libc6 2.10.1-0ubuntu16".
[19 May 2010 11:50] MySQL Verification Team
Thank you for the bug report. Repeatable on Slackware 13.1 X86_64:

*** glibc detected *** /home/miguel/bzr/mysql-5.1-main/sql/mysqld: double free or corruption (fasttop): 0x00000000011ac8e0 ***
======= Backtrace: =========
/lib64/libc.so.6(+0x76ce6)[0x7f51152acce6]
/lib64/libc.so.6(cfree+0x73)[0x7f51152b3553]
/home/miguel/bzr/mysql-5.1-main/sql/mysqld[0xb3a007]
/home/miguel/bzr/mysql-5.1-main/sql/mysqld[0xb39a73]
/home/miguel/bzr/mysql-5.1-main/sql/mysqld(_db_end_+0x15d)[0xb39d2e]
/home/miguel/bzr/mysql-5.1-main/sql/mysqld(my_end+0x341)[0xb054ee]
/home/miguel/bzr/mysql-5.1-main/sql/mysqld(main+0x63c)[0x69a95f]
/lib64/libc.so.6(__libc_start_main+0xfd)[0x7f5115254b6d]
/home/miguel/bzr/mysql-5.1-main/sql/mysqld[0x5990d9]
======= Memory map: ========
00400000-00e40000 r-xp 00000000 08:15 3298633                            /home/miguel/bzr/mysql-5.1-main/sql/mysqld
0103f000-01187000 rw-p 00a3f000 08:15 3298633                            /home/miguel/bzr/mysql-5.1-main/sql/mysqld
01187000-012ad000 rw-p 00000000 00:00 0                                  [heap]
7f5110000000-7f5110021000 rw-p 00000000 00:00 0 
7f5110021000-7f5114000000 ---p 00000000 00:00 0 
7f51147de000-7f51147df000 ---p 00000000 00:00 0 
7f51147df000-7f511481f000 rw-p 00000000 00:00 0 
7f511481f000-7f5114835000 r-xp 00000000 08:15 14444346                   /usr/lib64/libgcc_s.so.1
7f5114835000-7f5114a34000 ---p 00016000 08:15 14444346                   /usr/lib64/libgcc_s.so.1
7f5114a34000-7f5114a35000 rw-p 00015000 08:15 14444346                   /usr/lib64/libgcc_s.so.1
7f5114a35000-7f5114a36000 ---p 00000000 00:00 0 
7f5114a36000-7f5115236000 rw-p 00000000 00:00 0 
7f5115236000-7f51153a1000 r-xp 00000000 08:15 27000999                   /lib64/libc-2.11.1.so
7f51153a1000-7f51155a1000 ---p 0016b000 08:15 27000999                   /lib64/libc-2.11.1.so
7f51155a1000-7f51155a5000 r--p 0016b000 08:15 27000999                   /lib64/libc-2.11.1.so
7f51155a5000-7f51155a6000 rw-p 0016f000 08:15 27000999                   /lib64/libc-2.11.1.so
7f51155a6000-7f51155ab000 rw-p 00000000 00:00 0 
7f51155ab000-7f511562d000 r-xp 00000000 08:15 27000987                   /lib64/libm-2.11.1.so
7f511562d000-7f511582c000 ---p 00082000 08:15 27000987                   /lib64/libm-2.11.1.so
7f511582c000-7f511582d000 r--p 00081000 08:15 27000987                   /lib64/libm-2.11.1.so
7f511582d000-7f511582e000 rw-p 00082000 08:15 27000987                   /lib64/libm-2.11.1.so
7f511582e000-7f5115845000 r-xp 00000000 08:15 27000988                   /lib64/libnsl-2.11.1.so
7f5115845000-7f5115a44000 ---p 00017000 08:15 27000988                   /lib64/libnsl-2.11.1.so
7f5115a44000-7f5115a45000 r--p 00016000 08:15 27000988                   /lib64/libnsl-2.11.1.so
7f5115a45000-7f5115a46000 rw-p 00017000 08:15 27000988                   /lib64/libnsl-2.11.1.so
7f5115a46000-7f5115a48000 rw-p 00000000 00:00 0 
7f5115a48000-7f5115a51000 r-xp 00000000 08:15 27000985                   /lib64/libcrypt-2.11.1.so
7f5115a51000-7f5115c51000 ---p 00009000 08:15 27000985                   /lib64/libcrypt-2.11.1.so
7f5115c51000-7f5115c52000 r--p 00009000 08:15 27000985                   /lib64/libcrypt-2.11.1.so
7f5115c52000-7f5115c53000 rw-p 0000a000 08:15 27000985                   /lib64/libcrypt-2.11.1.so
7f5115c53000-7f5115c81000 rw-p 00000000 00:00 0 
7f5115c81000-7f5115c83000 r-xp 00000000 08:15 27001000                   /lib64/libdl-2.11.1.so
7f5115c83000-7f5115e83000 ---p 00002000 08:15 27001000                   /lib64/libdl-2.11.1.so
7f5115e83000-7f5115e84000 r--p 00002000 08:15 27001000                   /lib64/libdl-2.11.1.so
7f5115e84000-7f5115e85000 rw-p 00003000 08:15 27001000                   /lib64/libdl-2.11.1.so
7f5115e85000-7f5115e99000 r-xp 00000000 08:15 14418037                   /usr/lib64/libz.so.1.2.3
7f5115e99000-7f5116098000 ---p 00014000 08:15 14418037                   /usr/lib64/libz.so.1.2.3
7f5116098000-7f5116099000 rw-p 00013000 08:15 14418037                   /usr/lib64/libz.so.1.2.3
7f5116099000-7f51160b1000 r-xp 00000000 08:15 27000993                   /lib64/libpthread-2.11.1.so
7f51160b1000-7f51162b0000 ---p 00018000 08:15 27000993                   /lib64/libpthread-2.11.1.so
7f51162b0000-7f51162b1000 r--p 00017000 08:15 27000993                   /lib64/libpthread-2.11.1.so
7f51162b1000-7f51162b2000 rw-p 00018000 08:15 27000993                   /lib64/libpthread-2.11.1.so
7f51162b2000-7f51162b6000 rw-p 00000000 00:00 0 
7f51162b6000-7f51162d6000 r-xp 00000000 08:15 27001023                   /lib64/ld-2.11.1.so
7f51162e1000-7f51162e2000 ---p 00000000 00:00 0 
7f51162e2000-7f5116322000 rw-p 00000000 00:00 0 
7f5116322000-7f5116323000 ---p 00000000 00:00 0 
7f5116323000-7f5116363000 rw-p 00000000 00:00 0 
7f5116363000-7f5116364000 ---p 00000000 00:00 0 
7f5116364000-7f51163a4000 rw-p 00000000 00:00 0 
7f51164a5000-7f51164aa000 rw-p 00000000 00:00 0 
7f51164d4000-7f51164d5000 rw-p 00000000 00:00 0 
7f51164d5000-7f51164d6000 r--p 0001f000 08:15 27001023                   /lib64/ld-2.11.1.so
7f51164d6000-7f51164d7000 rw-p 00020000 08:15 27001023                   /lib64/ld-2.11.1.so
7f51164d7000-7f51164d8000 rw-p 00000000 00:00 0 
7fff75858000-7fff7586f000 rw-p 00000000 00:00 0                          [stack]
7fff759ff000-7fff75a00000 r-xp 00000000 00:00 0                          [vdso]
ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0                  [vsyscall]
------------------------------------------------------------
The servers were restarted 0 times
Spent 0.078 of 6 seconds executing testcases

Completed: All 1 tests were successful.

miguel@tikal:~/bzr/mysql-5.1-main/mysql-test$
[24 May 2010 16:48] Ryan Mack
It looks like the call to FreeState on init_settings was newly added to 5.1.47:

@@ -1510,7 +1517,10 @@ void _db_end_()
while ((discard= cs->stack))
{
if (discard == &init_settings)
+ {
+ FreeState (cs, discard, 0);
break;
+ }
cs->stack= discard->next;
FreeState(cs, discard, 1);
}

Later in the _db_end_() function is this (from a prior release):

tmp= init_settings;
... some stuff ...
FreeState(cs, &tmp, 0);

Because FreeState doesn't set the list head pointers to NULL, tmp contains pointers to the deleted first elements of the lists in init_settings.  The second call to FreeState ends up trying to free the same lists as the earlier call leading to glibc's double-free error.

My take on it is that the new call to FreeState on init_settings is unnecessary and should be removed. Commenting it out (returning to 5.1.46 version of _db_end_()) makes the crash go away. I feel I should add a disclaimer that there may be other reasons warranting the new call to FreeState that I missed.

-Ryan
[24 May 2010 16:57] Davi Arnaut
Closed as a duplicate of Bug#52884