Bug #29247 Double free in libmysqlclient_r when mysqld restarted
Submitted: 20 Jun 2007 17:46 Modified: 11 Jul 2007 22:37
Reporter: Andrew Agno Email Updates:
Status: Closed Impact on me:
None 
Category:MySQL Server: C API (client library) Severity:S3 (Non-critical)
Version:5.0.37, 5.0.41, 5.0.46-BK OS:Linux (Ubuntu 7.04 64 bit)
Assigned to: Alexey Botchkov CPU Architecture:Any
Tags: FREE, my_no_flags_free, mysql_real_connect, reconnect, segfault, segv

[20 Jun 2007 17:46] Andrew Agno
Description:
I get a segfault when I run a fairly simple program that makes a query of the form "SELECT count(*) from <table>", repeatedly.  On the server is a loop:
while [ 1 ]; do /etc/init.d/mysqld restart; sleep 5; done

Running under GDB, with the mysqlclient library built from scratch with debugging enabled, I get:
======= Backtrace: =========
/lib/libc.so.6[0x2b0b01160b23]
/lib/libc.so.6(cfree+0x8c)[0x2b0b0116426c]
/tmp/mysql-5.0.37-install/lib/mysql/libmysqlclient_r.so.15(my_no_flags_free+0x84)[0x2b0b001acdab]
/tmp/mysql-5.0.37-install/lib/mysql/libmysqlclient_r.so.15(mysql_real_connect+0x23e)[0x2b0b001ea601]
/tmp/mysql-5.0.37-install/lib/mysql/libmysqlclient_r.so.15(mysql_reconnect+0x230)[0x2b0b001ec3b1]
/tmp/mysql-5.0.37-install/lib/mysql/libmysqlclient_r.so.15(cli_advanced_command+0x7b)[0x2b0b001e7d69]
/tmp/mysql-5.0.37-install/lib/mysql/libmysqlclient_r.so.15(mysql_send_query+0x1a3)[0x2b0b001ed263]
/tmp/mysql-5.0.37-install/lib/mysql/libmysqlclient_r.so.15(mysql_real_query+0xc0)[0x2b0b001ed32e]
/tmp/reconnect(__gxx_personality_v0+0x35b)[0x4011ab]
/tmp/reconnect[0x4014b8]
/lib/libc.so.6(__libc_start_main+0xf4)[0x2b0b0110e8e4]
/tmp/reconnect(__gxx_personality_v0+0x59)[0x400ea9]

Also,
(gdb) where
#0  0x00002b0b01121cab in raise () from /lib/libc.so.6
#1  0x00002b0b01123660 in abort () from /lib/libc.so.6
#2  0x00002b0b0115966b in ?? () from /lib/libc.so.6
#3  0x00002b0b01160b23 in ?? () from /lib/libc.so.6
#4  0x00002b0b0116426c in free () from /lib/libc.so.6
#5  0x00002b0b001acdab in my_no_flags_free (ptr=0x6188d0 "") at my_malloc.c:59
#6  0x00002b0b001ea601 in mysql_real_connect (mysql=0x7fffaab46280, host=0x618860 "127.0.0.1", user=0x602bd0 "user", passwd=0x602bf0 "password", db=0x6188b0 "db", port=3306,
    unix_socket=0x0, client_flag=2147525261) at client.c:1826
#7  0x00002b0b001ec3b1 in mysql_reconnect (mysql=0x7fffaab46960) at client.c:2458
#8  0x00002b0b001e7d69 in cli_advanced_command (mysql=0x7fffaab46960, command=COM_QUERY, header=0x0, header_length=0, arg=0x618908 "SELECT count(*) from nodes", arg_length=26, skip_check=1 '\001',
    stmt=0x0) at client.c:672
#9  0x00002b0b001ed263 in mysql_send_query (mysql=0x7fffaab46960, query=0x618908 "SELECT count(*) from nodes", length=26) at client.c:2763
#10 0x00002b0b001ed32e in mysql_real_query (mysql=0x7fffaab46960, query=0x618908 "SELECT count(*) from nodes", length=26) at client.c:2774
#11 0x00000000004011ab in query (mysql=0x7fffaab46960, sql=@0x7fffaab46940) at TestReconnect.cpp:47
#12 0x00000000004014b8 in main (argc=1, argv=0x7fffaab46f48) at TestReconnect.cpp:70
(gdb) up 5
#5  0x00002b0b001acdab in my_no_flags_free (ptr=0x6188d0 "") at my_malloc.c:59
59          free(ptr);
Current language:  auto; currently c
(gdb) print ptr
$2 = 0x6188d0 ""

How to repeat:
Run the following program:
#include <mysql/mysql.h>
#include <iostream>

using namespace std;

string host ( "127.0.0.1" );
string username( "user" );
string password( "password" );
string database( "db" );
uint16_t port( 3306 );

bool connect( MYSQL* mysql )
{
    mysql_init( mysql );

    if ( !mysql_real_connect( mysql, host.c_str(), username.c_str(), password.c_str(), database.c_str(),
        port, NULL, 0 ) )
    {
        cerr << "Could not connect to database" << endl;
        return false;
    }

    if ( mysql_options( mysql, MYSQL_OPT_RECONNECT, "1" ) != 0 )
    {
        cerr << "Could not set database reconnect option" << endl;
        return false;
    }

    if ( mysql_options( mysql, MYSQL_READ_DEFAULT_FILE, "my.cnf" ) != 0 )
    {
        cerr << "Could not read client option file " << endl;
        return false;
    }

    if ( mysql_select_db( mysql, database.c_str() ) != 0)
    {
        cerr << "Could not select database" << endl;
        return false;
    }

    return true;
}

MYSQL_RES* query( MYSQL* mysql, string sql )
{
    if( mysql_real_query( mysql, sql.c_str(), sql.size() ) != 0 )
    {
        return NULL;
    }

    return mysql_store_result( mysql );
}

int main( int argc, char* argv[] )
{
    MYSQL mysql;

    if( !connect( &mysql ) )
    {
        return 1;
    }

    string sql( "SELECT count(*) from table" );

    for( size_t i =0;; ++i )
    {
        cout << ".";
        if( i % 180 == 0 ) cout << endl;
        MYSQL_RES* result = query( &mysql, sql );
        if ( result )
        {
            mysql_free_result( result );
        }
    }
}

Compile like:
g++ -g -o reconnect TestReconnect.cpp -I /path/to/mysql-5.0.37/include -L /path/to/mysql-5.0.37/lib/mysql -lmysqlclient_r -lz -lpthread

Run like:
LD_LIBRARY_PATH=/path/to/mysql-5.0.37/lib/mysql:$LD_LIBRARY_PATH gdb ./reconnect

On the mysql server side, start the loop given in the Description section.  The client should crash, eventually.

Suggested fix:
Unknown
[20 Jun 2007 19:42] Andrew Agno
This bug also occurs with 5.0.41
[21 Jun 2007 18:38] Valeriy Kravchuk
Thank you for a bug report. Verified just as described with latest 5.0.46-debug on SuSE Linux 9.3. I've got, eventually, while running your test code and continuosly restarting myslqd:

....................
openxs@linux:~/dbs/5.0> *** glibc detected *** double free or corruption (top): 0x08061510 ***
[25 Jun 2007 12:42] Bugs System
A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/29500

ChangeSet@1.2496, 2007-06-25 16:40:29+05:00, holyfoot@mysql.com +1 -0
  Bug #29247 Double free in libmysqlclient_r when mysql restarted.
  
  If one sets MYSQL_READ_DEFAULTS_FILE and MYSQL_READ_DEFAULT_GROUP options
  after mysql_real_connect() called with that MYSQL instance,
  these options will affect next mysql_reconnect then.
  As we use a copy of the original MYSQL object inside mysql_reconnect,
  and mysql_real_connect frees options.my_cnf_file and _group strings,
  we will free these twice when we execute mysql_reconnect with the
  same MYSQL for the second time.
  
  I don't think we should ever read defaults files handling mysql_reconnect.
  So i just set them to 0 for the temporary MYSQL object there/
[25 Jun 2007 12:50] Alexey Botchkov
I should note that manual says we should call mysql_options() before
the mysql_real_connect().
http://dev.mysql.com/doc/refman/5.1/en/mysql-options.html

And the reported problem appears as Andrew does it in opposite order
in his example.
[25 Jun 2007 17:54] Andrew Agno
Yes, moving around the order of calls also fixes this.  We had the order reversed because it didn't work for some older version of mysqlclient (see:
http://dev.mysql.com/doc/refman/5.0/en/mysql-options.html) and the comment stating:
'
Even though the documentation says that you should call mysql_options before mysql_real_connect, when setting MYSQL_OPT_RECONNECT you MUST do so after the connection has been successfully established.

This is true as late as 5.0.16.
'
[1 Jul 2007 19:57] Bugs System
Pushed into 5.1.21-beta
[1 Jul 2007 20:02] Bugs System
Pushed into 5.0.46
[11 Jul 2007 22:37] Paul Dubois
Noted in 5.0.46, 5.1.21 changelogs.

Calling mysql_options() after mysql_real_connect() could cause 
clients to crash.

Also updated the mysql_options() pages to mention the problem
of mysql_real_connect() resetting auto-reconnect behavior prior
to MySQL 5.0.19/5.1.6.