Bug #84147 isValid intermittent segfault
Submitted: 9 Dec 2016 11:40 Modified: 16 Dec 2016 11:26
Reporter: Martin O'Neal Email Updates:
Status: Verified Impact on me:
None 
Category:Connector / C++ Severity:S3 (Non-critical)
Version:libmysqlclient.so.20.3.3 OS:Ubuntu (Linux version 4.4.0-45-generic (buildd@lgw01-34) (gcc version 5.4.0 20160609 (Ubuntu 5.4.0-6ubuntu1~)
Assigned to: CPU Architecture:Any

[9 Dec 2016 11:40] Martin O'Neal
Description:

I get intermittent segfaults when calling isValid, but only if the connection has timed-out.

mysql.log:
2016-12-08 22:44:14 26960 [Warning] Aborted connection 4197 to db: 'vsdashboard' user: 'vsdashboard' host: '172.28.0.187' (Got timeout reading communication packets)

kern.log:

segfault at 7f51b1237030 ip 00007f51b86f1b86 sp 00007f51b1b84590 error 4 in libmysqlclient.so.20.3.3[7f51b86be000+393000]

valgrind:

==27573== Invalid read of size 1
==27573==    at 0x7379B86: ??? (in /usr/lib/x86_64-linux-gnu/libmysqlclient.so.20.3.3)
==27573==    by 0x736CB65: mysql_ping (in /usr/lib/x86_64-linux-gnu/libmysqlclient.so.20.3.3)
==27573==    by 0x4F198A9: sql::mysql::NativeAPI::LibmysqlStaticProxy::ping(st_mysql*) (in /usr/lib/libmysqlcppconn.so.7.1.1.7)
==27573==    by 0x4F1B720: sql::mysql::NativeAPI::MySQL_NativeConnectionWrapper::ping() (in /usr/lib/libmysqlcppconn.so.7.1.1.7)
==27573==    by 0x4EC3F5E: sql::mysql::MySQL_Connection::isValid() (in /usr/lib/libmysqlcppconn.so.7.1.1.7)
==27573==    by 0x48C6FF: UdMysql::openConnection() (in /opt/Asparagus/bin/Asparagus)
==27573==    by 0x40E529: Asparagus::masterThread() (in /opt/Asparagus/bin/Asparagus)
==27573==    by 0x40A77F: main (in /opt/Asparagus/bin/Asparagus)
==27573==  Address 0xd9430d0 is 32 bytes inside a block of size 8,231 free'd
==27573==    at 0x4C2EDEB: free (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==27573==    by 0x737D894: ??? (in /usr/lib/x86_64-linux-gnu/libmysqlclient.so.20.3.3)
==27573==    by 0x7372E1E: ??? (in /usr/lib/x86_64-linux-gnu/libmysqlclient.so.20.3.3)
==27573==    by 0x73730B7: ??? (in /usr/lib/x86_64-linux-gnu/libmysqlclient.so.20.3.3)
==27573==    by 0x73799AA: ??? (in /usr/lib/x86_64-linux-gnu/libmysqlclient.so.20.3.3)
==27573==    by 0x736CB65: mysql_ping (in /usr/lib/x86_64-linux-gnu/libmysqlclient.so.20.3.3)
==27573==    by 0x4F198A9: sql::mysql::NativeAPI::LibmysqlStaticProxy::ping(st_mysql*) (in /usr/lib/libmysqlcppconn.so.7.1.1.7)
==27573==    by 0x4F1B720: sql::mysql::NativeAPI::MySQL_NativeConnectionWrapper::ping() (in /usr/lib/libmysqlcppconn.so.7.1.1.7)
==27573==    by 0x4EC3F5E: sql::mysql::MySQL_Connection::isValid() (in /usr/lib/libmysqlcppconn.so.7.1.1.7)
==27573==    by 0x48C6FF: UdMysql::openConnection() (in /opt/Asparagus/bin/Asparagus)
==27573==    by 0x40E529: Asparagus::masterThread() (in /opt/Asparagus/bin/Asparagus)
==27573==    by 0x40A77F: main (in /opt/Asparagus/bin/Asparagus)
==27573==  Block was alloc'd at
==27573==    at 0x4C2DB8F: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==27573==    by 0x73A8AF7: ??? (in /usr/lib/x86_64-linux-gnu/libmysqlclient.so.20.3.3)
==27573==    by 0x737D7B5: ??? (in /usr/lib/x86_64-linux-gnu/libmysqlclient.so.20.3.3)
==27573==    by 0x73777DB: mysql_real_connect (in /usr/lib/x86_64-linux-gnu/libmysqlclient.so.20.3.3)
==27573==    by 0x4F198F4: sql::mysql::NativeAPI::LibmysqlStaticProxy::real_connect(st_mysql*, char const*, char const*, char const*, char const*, un
signed int, char const*, unsigned long) (in /usr/lib/libmysqlcppconn.so.7.1.1.7)
==27573==    by 0x4F1AECF: sql::mysql::NativeAPI::MySQL_NativeConnectionWrapper::connect(sql::SQLString const&, sql::SQLString const&, sql::SQLString
 const&, sql::SQLString const&, unsigned int, sql::SQLString const&, unsigned long) (in /usr/lib/libmysqlcppconn.so.7.1.1.7)
==27573==    by 0x4EBE017: sql::mysql::MySQL_Connection::init(std::map<sql::SQLString, sql::Variant, std::less<sql::SQLString>, std::allocator<std::p
air<sql::SQLString const, sql::Variant> > >&) (in /usr/lib/libmysqlcppconn.so.7.1.1.7)
==27573==    by 0x4EBB820: sql::mysql::MySQL_Connection::MySQL_Connection(sql::Driver*, sql::mysql::NativeAPI::NativeConnectionWrapper&, std::map<sql
::SQLString, sql::Variant, std::less<sql::SQLString>, std::allocator<std::pair<sql::SQLString const, sql::Variant> > >&) (in /usr/lib/libmysqlcppconn
.so.7.1.1.7)
==27573==    by 0x4ECE92C: sql::mysql::MySQL_Driver::connect(std::map<sql::SQLString, sql::Variant, std::less<sql::SQLString>, std::allocator<std::pa
ir<sql::SQLString const, sql::Variant> > >&) (in /usr/lib/libmysqlcppconn.so.7.1.1.7)
==27573==    by 0x48CD10: UdMysql::openConnection() (in /opt/Asparagus/bin/Asparagus)
==27573==    by 0x463DDF: UdConfiguration::getSystem(std::unordered_map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>
 >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::hash<std::__cxx11::basic_string<char, std::char_traits<char
>, std::allocator<char> > >, std::equal_to<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pai
r<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std
::allocator<char> > > > >&) (in /opt/Asparagus/bin/Asparagus)
==27573==    by 0x40D25F: Asparagus::masterThread() (in /opt/Asparagus/bin/Asparagus)
==27573==    by 0x40A77F: main (in /opt/Asparagus/bin/Asparagus)

How to repeat:
Set the wait timeout low so that the connection expires predictably:

set global wait_timeout=10;

Loop through the following snippet within a valgrind session and the isValid call will fail intermittently if the connection has timed-out.

sql::ConnectOptionsMap connectionOptions;
bool trueOption = true;

connectionOptions[ "hostName" ] = host_;  
connectionOptions[ "userName" ] = user_;  
connectionOptions[ "password" ] = password_;  
connectionOptions[ "OPT_CONNECT_TIMEOUT" ] = ( int ) connectionTimeoutSeconds_;  
connectionOptions[ "OPT_READ_TIMEOUT" ] = ( int ) readWriteTimeoutSeconds_;  
connectionOptions[ "OPT_WRITE_TIMEOUT" ] = ( int ) readWriteTimeoutSeconds_;  
connectionOptions[ "sslEnforce" ] = isTlsRequired_;  
connectionOptions[ "sslVerify" ] = isTlsRequired_;  
connectionOptions[ "sslCipher" ] = tlsCipherSuite_;  
connectionOptions[ "SET_CHARSET_NAME" ] = characterSet_;  
connectionOptions[ "CLIENT_IGNORE_SIGPIPE" ] = trueOption;  
connectionOptions[ "schema" ] = database_; 

// check if connection has failed
if( connection_ != NULL )
    if( ! connection_->isValid( ) && ! connection_->reconnect( ) )
    {
        connection_->close( );
        delete connection_;
        connection_ = NULL;
    }

// check if connection is not already open
if( connection_ == NULL )
{
    // connect to server
    connection_ = driver_->connect( connectionOptions );

    // set connection to manual commit
    connection_->setAutoCommit( false );
}

sleep( 15 );

valgrind command line:

/usr/bin/valgrind --log-file=/tmp/valgrind.txt --track-origins=yes --tool=memcheck --verbose --num-callers=50 --show-reachable=yes --leak-check=full --trace-children=yes /opt/asparagus/bin/asparagus

Suggested fix:

It looks like a logic failure in the code, where a buffer is free'd before it is accessed...
[10 Dec 2016 9:00] Martin O'Neal
As a workaround, isValid can be called periodically on an idle connection, prior to it timing out...
[16 Dec 2016 11:26] Chiranjeevi Battula
Hello Martin O'Neal,

Thank you for the bug report.
Verified based on internal discussion with dev's.

Thanks,
Chiranjeevi.