Description:
I get intermittent segfaults when calling isValid, but only if the connection has timed-out.
mysql.log:
2016-12-08 22:44:14 26960 [Warning] Aborted connection 4197 to db: 'vsdashboard' user: 'vsdashboard' host: '172.28.0.187' (Got timeout reading communication packets)
kern.log:
segfault at 7f51b1237030 ip 00007f51b86f1b86 sp 00007f51b1b84590 error 4 in libmysqlclient.so.20.3.3[7f51b86be000+393000]
valgrind:
==27573== Invalid read of size 1
==27573== at 0x7379B86: ??? (in /usr/lib/x86_64-linux-gnu/libmysqlclient.so.20.3.3)
==27573== by 0x736CB65: mysql_ping (in /usr/lib/x86_64-linux-gnu/libmysqlclient.so.20.3.3)
==27573== by 0x4F198A9: sql::mysql::NativeAPI::LibmysqlStaticProxy::ping(st_mysql*) (in /usr/lib/libmysqlcppconn.so.7.1.1.7)
==27573== by 0x4F1B720: sql::mysql::NativeAPI::MySQL_NativeConnectionWrapper::ping() (in /usr/lib/libmysqlcppconn.so.7.1.1.7)
==27573== by 0x4EC3F5E: sql::mysql::MySQL_Connection::isValid() (in /usr/lib/libmysqlcppconn.so.7.1.1.7)
==27573== by 0x48C6FF: UdMysql::openConnection() (in /opt/Asparagus/bin/Asparagus)
==27573== by 0x40E529: Asparagus::masterThread() (in /opt/Asparagus/bin/Asparagus)
==27573== by 0x40A77F: main (in /opt/Asparagus/bin/Asparagus)
==27573== Address 0xd9430d0 is 32 bytes inside a block of size 8,231 free'd
==27573== at 0x4C2EDEB: free (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==27573== by 0x737D894: ??? (in /usr/lib/x86_64-linux-gnu/libmysqlclient.so.20.3.3)
==27573== by 0x7372E1E: ??? (in /usr/lib/x86_64-linux-gnu/libmysqlclient.so.20.3.3)
==27573== by 0x73730B7: ??? (in /usr/lib/x86_64-linux-gnu/libmysqlclient.so.20.3.3)
==27573== by 0x73799AA: ??? (in /usr/lib/x86_64-linux-gnu/libmysqlclient.so.20.3.3)
==27573== by 0x736CB65: mysql_ping (in /usr/lib/x86_64-linux-gnu/libmysqlclient.so.20.3.3)
==27573== by 0x4F198A9: sql::mysql::NativeAPI::LibmysqlStaticProxy::ping(st_mysql*) (in /usr/lib/libmysqlcppconn.so.7.1.1.7)
==27573== by 0x4F1B720: sql::mysql::NativeAPI::MySQL_NativeConnectionWrapper::ping() (in /usr/lib/libmysqlcppconn.so.7.1.1.7)
==27573== by 0x4EC3F5E: sql::mysql::MySQL_Connection::isValid() (in /usr/lib/libmysqlcppconn.so.7.1.1.7)
==27573== by 0x48C6FF: UdMysql::openConnection() (in /opt/Asparagus/bin/Asparagus)
==27573== by 0x40E529: Asparagus::masterThread() (in /opt/Asparagus/bin/Asparagus)
==27573== by 0x40A77F: main (in /opt/Asparagus/bin/Asparagus)
==27573== Block was alloc'd at
==27573== at 0x4C2DB8F: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==27573== by 0x73A8AF7: ??? (in /usr/lib/x86_64-linux-gnu/libmysqlclient.so.20.3.3)
==27573== by 0x737D7B5: ??? (in /usr/lib/x86_64-linux-gnu/libmysqlclient.so.20.3.3)
==27573== by 0x73777DB: mysql_real_connect (in /usr/lib/x86_64-linux-gnu/libmysqlclient.so.20.3.3)
==27573== by 0x4F198F4: sql::mysql::NativeAPI::LibmysqlStaticProxy::real_connect(st_mysql*, char const*, char const*, char const*, char const*, un
signed int, char const*, unsigned long) (in /usr/lib/libmysqlcppconn.so.7.1.1.7)
==27573== by 0x4F1AECF: sql::mysql::NativeAPI::MySQL_NativeConnectionWrapper::connect(sql::SQLString const&, sql::SQLString const&, sql::SQLString
const&, sql::SQLString const&, unsigned int, sql::SQLString const&, unsigned long) (in /usr/lib/libmysqlcppconn.so.7.1.1.7)
==27573== by 0x4EBE017: sql::mysql::MySQL_Connection::init(std::map<sql::SQLString, sql::Variant, std::less<sql::SQLString>, std::allocator<std::p
air<sql::SQLString const, sql::Variant> > >&) (in /usr/lib/libmysqlcppconn.so.7.1.1.7)
==27573== by 0x4EBB820: sql::mysql::MySQL_Connection::MySQL_Connection(sql::Driver*, sql::mysql::NativeAPI::NativeConnectionWrapper&, std::map<sql
::SQLString, sql::Variant, std::less<sql::SQLString>, std::allocator<std::pair<sql::SQLString const, sql::Variant> > >&) (in /usr/lib/libmysqlcppconn
.so.7.1.1.7)
==27573== by 0x4ECE92C: sql::mysql::MySQL_Driver::connect(std::map<sql::SQLString, sql::Variant, std::less<sql::SQLString>, std::allocator<std::pa
ir<sql::SQLString const, sql::Variant> > >&) (in /usr/lib/libmysqlcppconn.so.7.1.1.7)
==27573== by 0x48CD10: UdMysql::openConnection() (in /opt/Asparagus/bin/Asparagus)
==27573== by 0x463DDF: UdConfiguration::getSystem(std::unordered_map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>
>, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::hash<std::__cxx11::basic_string<char, std::char_traits<char
>, std::allocator<char> > >, std::equal_to<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pai
r<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std
::allocator<char> > > > >&) (in /opt/Asparagus/bin/Asparagus)
==27573== by 0x40D25F: Asparagus::masterThread() (in /opt/Asparagus/bin/Asparagus)
==27573== by 0x40A77F: main (in /opt/Asparagus/bin/Asparagus)
How to repeat:
Set the wait timeout low so that the connection expires predictably:
set global wait_timeout=10;
Loop through the following snippet within a valgrind session and the isValid call will fail intermittently if the connection has timed-out.
sql::ConnectOptionsMap connectionOptions;
bool trueOption = true;
connectionOptions[ "hostName" ] = host_;
connectionOptions[ "userName" ] = user_;
connectionOptions[ "password" ] = password_;
connectionOptions[ "OPT_CONNECT_TIMEOUT" ] = ( int ) connectionTimeoutSeconds_;
connectionOptions[ "OPT_READ_TIMEOUT" ] = ( int ) readWriteTimeoutSeconds_;
connectionOptions[ "OPT_WRITE_TIMEOUT" ] = ( int ) readWriteTimeoutSeconds_;
connectionOptions[ "sslEnforce" ] = isTlsRequired_;
connectionOptions[ "sslVerify" ] = isTlsRequired_;
connectionOptions[ "sslCipher" ] = tlsCipherSuite_;
connectionOptions[ "SET_CHARSET_NAME" ] = characterSet_;
connectionOptions[ "CLIENT_IGNORE_SIGPIPE" ] = trueOption;
connectionOptions[ "schema" ] = database_;
// check if connection has failed
if( connection_ != NULL )
if( ! connection_->isValid( ) && ! connection_->reconnect( ) )
{
connection_->close( );
delete connection_;
connection_ = NULL;
}
// check if connection is not already open
if( connection_ == NULL )
{
// connect to server
connection_ = driver_->connect( connectionOptions );
// set connection to manual commit
connection_->setAutoCommit( false );
}
sleep( 15 );
valgrind command line:
/usr/bin/valgrind --log-file=/tmp/valgrind.txt --track-origins=yes --tool=memcheck --verbose --num-callers=50 --show-reachable=yes --leak-check=full --trace-children=yes /opt/asparagus/bin/asparagus
Suggested fix:
It looks like a logic failure in the code, where a buffer is free'd before it is accessed...