Bug #19474 readline bug: mysql: free(): invalid pointer

Description:

I used a fairly normal Sql query, just a straight select, but the request crashed, although the server kept running ok.

Initially I used it from a PHP script, which after a long time would respond "Mysql server has gone away", but then I tried to debug the situation, my using commandline mysql, version 5.0.18 the result was as below.

The query is on three fairly simple tables, and I have been using this query for quite a long time in my php application, phplist (www.phplist.com) starting with Mysql 3 and now mostly using Mysql 4 in production situations. Only recently I started using Mysql 5 for my development workstation and was actually trying to reproduce a bug in my own app, due to low memory, when this happened. I have 750Mb of memory.

The sql query is "select distinct user.id from phplist_listuser as listuser, phplist_user_user as user, phplist_listmessage as listmessage where listmessage.messageid = 40 and listmessage.listid = listuser.listid and user.id = listuser.userid and user.confirmed and !user.blacklisted and listuser.userid not in (XXX)"

where XXX is a growing list of "userIDs" for users that have received the message from the newsletter application before. The error started to happen when XXX had grown quite large. 

The actual query is quite large, so I've put it online on http://tincan.co.uk/mysqlcrash.txt because it would be too big to post in this bugreport. But you don't really need it, because it's system dependent, but below in "how to reproduce" I've worked out a way to reproduce the problem.

The query takes quite a while and then comes back with 

*** glibc detected *** mysql: free(): invalid pointer: 0x09a014f8 ***
======= Backtrace: =========
/lib/libc.so.6[0x867f18]
/lib/libc.so.6(__libc_free+0x79)[0x86b41d]
mysql(rl_free_undo_list+0x1b)[0x8069b5b]
mysql(readline_internal_teardown+0xad)[0x805b83d]
mysql(readline+0x5e)[0x805ba3e]
mysql[0x8057295]
mysql(main+0x4b8)[0x8058c38]
/lib/libc.so.6(__libc_start_main+0xdc)[0x8197e4]
mysql[0x8051b41]
======= Memory map: ========
00123000-0023c000 r-xp 00000000 fd:00 8749057    /usr/lib/mysql/libmysqlclient.so.15.0.0
0023c000-00267000 rwxp 00119000 fd:00 8749057    /usr/lib/mysql/libmysqlclient.so.15.0.0
0027d000-00286000 r-xp 00000000 fd:00 720937     /lib/libnss_files-2.4.so
00286000-00287000 r-xp 00008000 fd:00 720937     /lib/libnss_files-2.4.so
00287000-00288000 rwxp 00009000 fd:00 720937     /lib/libnss_files-2.4.so
005db000-005ea000 r-xp 00000000 fd:00 720965     /lib/libresolv-2.4.so
005ea000-005eb000 r-xp 0000e000 fd:00 720965     /lib/libresolv-2.4.so
005eb000-005ec000 rwxp 0000f000 fd:00 720965     /lib/libresolv-2.4.so
005ec000-005ee000 rwxp 005ec000 00:00 0
005f0000-00601000 r-xp 00000000 fd:00 722880     /lib/libnsl-2.4.so
00601000-00602000 r-xp 00010000 fd:00 722880     /lib/libnsl-2.4.so
00602000-00603000 rwxp 00011000 fd:00 722880     /lib/libnsl-2.4.so
00603000-00605000 rwxp 00603000 00:00 0
0066a000-00682000 r-xp 00000000 fd:00 4149937    /usr/lib/libgssapi_krb5.so.2.2
00682000-00683000 rwxp 00017000 fd:00 4149937    /usr/lib/libgssapi_krb5.so.2.2
00711000-00735000 r-xp 00000000 fd:00 4138582    /usr/lib/libk5crypto.so.3.0
00735000-00736000 rwxp 00024000 fd:00 4138582    /usr/lib/libk5crypto.so.3.0
00765000-0076a000 r-xp 00000000 fd:00 722881     /lib/libcrypt-2.4.so
0076a000-0076b000 r-xp 00004000 fd:00 722881     /lib/libcrypt-2.4.so
0076b000-0076c000 rwxp 00005000 fd:00 722881     /lib/libcrypt-2.4.so
0076c000-00793000 rwxp 0076c000 00:00 0
007b7000-007ba000 r-xp 00000000 fd:00 4149142    /usr/lib/libkrb5support.so.0.0
007ba000-007bb000 rwxp 00002000 fd:00 4149142    /usr/lib/libkrb5support.so.0.0
007e6000-007e7000 r-xp 007e6000 00:00 0          [vdso]
007e7000-00800000 r-xp 00000000 fd:00 722856     /lib/ld-2.4.so
00800000-00801000 r-xp 00018000 fd:00 722856     /lib/ld-2.4.so
00801000-00802000 rwxp 00019000 fd:00 722856     /lib/ld-2.4.so
00804000-00930000 r-xp 00000000 fd:00 722868     /lib/libc-2.4.so
00930000-00933000 r-xp 0012b000 fd:00 722868     /lib/libc-2.4.so
00933000-00934000 rwxp 0012e000 fd:00 722868     /lib/libc-2.4.so
00934000-00937000 rwxp 00934000 00:00 0
00939000-0095c000 r-xp 00000000 fd:00 722869     /lib/libm-2.4.so
0095c000-0095d000 r-xp 00022000 fd:00 722869     /lib/libm-2.4.so
0095d000-0095e000 rwxp 00023000 fd:00 722869     /lib/libm-2.4.so
00960000-00962000 r-xp 00000000 fd:00 722870     /lib/libdl-2.4.so
00962000-00963000 r-xp 00001000 fd:00 722870     /lib/libdl-2.4.so
00963000-00964000 rwxp 00002000 fd:00 722870     /lib/libdl-2.4.so
00966000-00978000 r-xp 00000000 fd:00 4153180    /usr/lib/libz.so.1.2.3
00978000-00979000 rwxp 00011000 fd:00 4153180    /usr/lib/libz.so.1.2.3
00bfb000-00c06000 r-xp 00000000 fd:00 722873     /lib/libgcc_s-4.1.0-20060304.so.1
00c06000-00c07000 rwxp 0000a000 fd:00 722873     /lib/libgcc_s-4.1.0-20060304.so.1
00c9d000-00d7f000 r-xp 00000000 fd:00 4153426    /usr/lib/libstdc++.so.6.0.8
00d7f000-00d83000 r-xp 000e2000 fd:00 4153426    /usr/lib/libstdc++.so.6.0.8
00d83000-00d84000 rwxp 000e6000 fd:00 4153426    /usr/lib/libstdc++.so.6.0.8
00d84000-00d8a000 rwxp 00d84000 00:00 0
00df7000-00df9000 r-xp 00000000 fd:00 722876     /lib/libcom_err.so.2.1
00df9000-00dfa000 rwxp 00001000 fd:00 722876     /lib/libcom_err.so.2.1
02000000-0211f000 r-xp 00000000 fd:00 720931     /lib/libcrypto.so.0.9.8a
0211f000-02132000 rwxp 0011e000 fd:00 720931     /lib/libcrypto.so.0.9.8a
02132000-02135000 rwxp 02132000 00:00 0
02137000-02178000 r-xp 00000000 fd:00 720945     /lib/libssl.so.0.9.8a
02178000-0217c000 rwxp 00040000 fd:00 720945     /lib/libssl.so.0.9.8a
04fa5000-04fe5000 r-xp 00000000Aborted

How to repeat:

the following PHP script crashes it for me every single time I run it. My workstation is unresponsive, or at least rather slow at responding, for quite a while, but it always ends with "Mysql Error: 2013 Lost connection to MySQL server during query".

That's only testing with 10k "users". I have several systems running 80k+ users and I'd like it work work with unlimited users preferably.

<?

$db = mysql_connect('localhost','root','');
mysql_query('drop database crashtestdb');
mysql_query('create database crashtestdb',$db);
mysql_select_db('crashtestdb');

mysql_query('create table phplist_user_user (id integer not null primary key auto_increment, confirmed integer default 1, blacklisted integer default 0);');
mysql_query('create table phplist_listuser (listid integer not null, userid integer not null, primary key (listid, userid));');
mysql_query('create table phplist_listmessage (listid integer not null, messageid integer not null, primary key (listid,messageid));');
$userids = array();
for ($i=1;$i<10000;$i++) {
  mysql_query('insert into phplist_user_user (confirmed,blacklisted) values(1,0)');
  $id = mysql_insert_id();
  mysql_query('insert into phplist_listuser (listid,userid) values(1,'.$id.')');
  array_push($userids,$id);
}
for ($i=1;$i<50;$i++) {
  mysql_query('insert into phplist_listmessage (listid,messageid) values(1,'.$i.')');
}

# take off a few, so there's some left to select
for ($i=0;$i<20;$i++) {
  array_pop($userids);
}
$done = join(',',$userids);

# do the fatal query
$req = mysql_query('select distinct user.id from phplist_listuser as listuser, phplist_user_user as user, phplist_listmessage as listmessage where listmessage.messageid = 1 and listmessage.listid = listuser.listid and user.id = listuser.userid and user.confirmed and !user.blacklisted and listuser.userid not in ('.$done.')');
$errno = mysql_errno();

if ($errno) {
  print "Mysql Error: $errno ". mysql_error();
}

Thank you for a problem report. Please, try to repeat with a newer version, 5.0.21. Your report is very similar to bug #15872, fixed in 5.0.21.

yes, you're right, it does sound like 15872

updated my Mysql using yum on the Fedora Test Repo, where 5.0.21 has been submitted.

When I run my "crashit" php script, it now returns the correct values. 

----
Doing query
No Error
9980 9981 9982 9983 9984 9985 9986 9987 9988 9989 9990 9991 9992 9993 9994 9995 9996 9997 9998 9999
----

However, when I pasted the query in the mysql console, being connected from localhost to server on localhost, I still got the crash. (Output pasted below).

And strangely enough, when I put the query in a text file and pipe it into the mysql commandline ("mysql < crash.sql") it returns fine with the values.

So, it's kind of fixed but not entirely. It's less critical though, because someone is unlikely to type such a query in console, and more likely to use connectors, but the crash is still reproducable.

-------- output from console, when using long NOT IN () ---------

*** glibc detected *** mysql: free(): invalid pointer: 0x0869a570 ***
======= Backtrace: =========
/lib/libc.so.6[0xc4cf18]
/lib/libc.so.6(__libc_free+0x79)[0xc5041d]
mysql(rl_free_undo_list+0x1b)[0x8069d6b]
mysql(readline_internal_teardown+0xad)[0x805ba4d]
mysql(readline+0x5e)[0x805bc4e]
mysql[0x8057505]
mysql(main+0x4b8)[0x8058e48]
/lib/libc.so.6(__libc_start_main+0xdc)[0xbfe7e4]
mysql[0x8051b71]
======= Memory map: ========
0010b000-0011c000 r-xp 00000000 fd:00 11011157   /lib/libnsl-2.4.so
0011c000-0011d000 r-xp 00010000 fd:00 11011157   /lib/libnsl-2.4.so
0011d000-0011e000 rwxp 00011000 fd:00 11011157   /lib/libnsl-2.4.so
0011e000-00120000 rwxp 0011e000 00:00 0
0016f000-001af000 r-xp 00000000 fd:00 13083761   /usr/lib/libncurses.so.5.5
001af000-001b7000 rwxp 00040000 fd:00 13083761   /usr/lib/libncurses.so.5.5
001b7000-001b8000 rwxp 001b7000 00:00 0
00216000-00225000 r-xp 00000000 fd:00 11010073   /lib/libresolv-2.4.so
00225000-00226000 r-xp 0000e000 fd:00 11010073   /lib/libresolv-2.4.so
00226000-00227000 rwxp 0000f000 fd:00 11010073   /lib/libresolv-2.4.so
00227000-00229000 rwxp 00227000 00:00 0
0022b000-0022d000 r-xp 00000000 fd:00 11010077   /lib/libcom_err.so.2.1
0022d000-0022e000 rwxp 00001000 fd:00 11010077   /lib/libcom_err.so.2.1
00230000-00233000 r-xp 00000000 fd:00 13093730   /usr/lib/libkrb5support.so.0.0
00233000-00234000 rwxp 00002000 fd:00 13093730   /usr/lib/libkrb5support.so.0.0
00242000-00361000 r-xp 00000000 fd:00 11011169   /lib/libcrypto.so.0.9.8a
00361000-00374000 rwxp 0011e000 fd:00 11011169   /lib/libcrypto.so.0.9.8a
00374000-00377000 rwxp 00374000 00:00 0
00573000-005e6000 r-xp 00000000 fd:00 13094402   /usr/lib/libkrb5.so.3.2
005e6000-005e8000 rwxp 00073000 fd:00 13094402   /usr/lib/libkrb5.so.3.2
005ea000-00602000 r-xp 00000000 fd:00 13094403   /usr/lib/libgssapi_krb5.so.2.2
00602000-00603000 rwxp 00017000 fd:00 13094403   /usr/lib/libgssapi_krb5.so.2.2
00605000-00629000 r-xp 00000000 fd:00 13094401   /usr/lib/libk5crypto.so.3.0
00629000-0062a000 rwxp 00024000 fd:00 13094401   /usr/lib/libk5crypto.so.3.0
006c4000-00705000 r-xp 00000000 fd:00 11010079   /lib/libssl.so.0.9.8a
00705000-00709000 rwxp 00040000 fd:00 11010079   /lib/libssl.so.0.9.8a
00883000-0088c000 r-xp 00000000 fd:00 11010089   /lib/libnss_files-2.4.so
0088c000-0088d000 r-xp 00008000 fd:00 11010089   /lib/libnss_files-2.4.so
0088d000-0088e000 rwxp 00009000 fd:00 11010089   /lib/libnss_files-2.4.so
00bcb000-00bcc000 r-xp 00bcb000 00:00 0          [vdso]
00bcc000-00be5000 r-xp 00000000 fd:00 11011150   /lib/ld-2.4.so
00be5000-00be6000 r-xp 00018000 fd:00 11011150   /lib/ld-2.4.so
00be6000-00be7000 rwxp 00019000 fd:00 11011150   /lib/ld-2.4.so
00be9000-00d15000 r-xp 00000000 fd:00 11011151   /lib/libc-2.4.so
00d15000-00d18000 r-xp 0012b000 fd:00 11011151   /lib/libc-2.4.so
00d18000-00d19000 rwxp 0012e000 fd:00 11011151   /lib/libc-2.4.so
00d19000-00d1c000 rwxp 00d19000 00:00 0
00d1e000-00d41000 r-xp 00000000 fd:00 11011154   /lib/libm-2.4.so
00d41000-00d42000 r-xp 00022000 fd:00 11011154   /lib/libm-2.4.so
00d42000-00d43000 rwxp 00023000 fd:00 11011154   /lib/libm-2.4.so
00d45000-00d47000 r-xp 00000000 fd:00 11011153   /lib/libdl-2.4.so
00d47000-00d48000 r-xp 00001000 fd:00 11011153   /lib/libdl-2.4.so
00d48000-00d49000 rwxp 00002000 fd:00 11011153   /lib/libdl-2.4.so
00d4b000-00d5d000 r-xp 00000000 fd:00 13093021   /usr/lib/libz.so.1.2.3
00d5d000-00d5e000 rwxp 00011000 fd:00 13093021   /usr/lib/libz.so.1.2.3
00e58000-00f77000 r-xp 00000000 fd:00 14156160   /usr/lib/mysql/libmysqlclient.so.15.0.0
00f77000-00fb9000 rwxp 0011e000 fd:00 14156160   /usr/lib/mysql/libmysqlclient.so.15.0.0
00fb9000-00fba000 rwxp 00fb9000 00:00 0
058f8000-05903000 r-xp 00000000 fd:00 11011155   /lib/libgcc_s-4.1.0-20060304.so.1
05903000-05904000 rwxp 0000a000 fd:00 11011155   /lib/libgcc_s-4.1.0-20060304.so.1
05906000-059e8000 r-xp 00000000 fd:00 13085639   /usr/lib/libstdc++.so.6.0.8
059e8000-059ec000 r-xp 000e2000 fd:00 13085639   /usr/lib/libstdc++.so.6.0.8
059ec000-059ed000 rwxp 000e6000 fd:00 13085639   /usr/lib/libstdc++.so.6.0.8
059ed000-059f3000 rwxp 059ed000 00:00 0
05a3d000-05a42000 r-xp 00000000 fd:00 11011164   /lib/libcrypt-2.4.so
05a42000-05a43000 r-xp 00004000 fd:00 11011164   /lib/libcrypt-2.4.so
05a43000-05a44000 rwxp 00005000 fd:00 11011164   /lib/libcrypt-2.4.so
05a44000-05a6b000 rwxp 05a44000 00:00 0
08048000-0807d000 r-xp 00000000 fd:00 13078688   /usr/bin/mysql
0807d000-08082000 rw-p 00034000 fd:00 13078688   /usr/bin/mysql
08082000-08084000 rw-p 08082000 00:00 0
0866c000-086f9000 rw-p 0866c000 00:00 0          [heap]
b7b00000-b7b21000 rw-p b7b00000 00:00 0
b7b21000-b7c00000 ---p b7b21000 00:00 0
b7d00000-b7f00000 r--p 00000000 fd:00 13077720   /usr/lib/locale/locale-archive
b7f00000-b7f05000 rw-p b7f00000 00:00 0
b7f10000-b7f17000 r--s 00000000 fd:00 13172995   /usr/lib/gconv/gconv-modules.cache
b7f17000-b7f19000 rw-p b7f17000 00:00 0
bfa02000-bfa18000 rw-p bfa02000 00:00 0          [stack]
Aborted

So, it looks like a mysql command line client related bug. Moreover, it can be readline-related. Do you agree?

Anyway, please, specify the exact MySQL's binaries package used (we have no FC5-specific) or, if you compiled 5.0.21 from sources, provide the exact configure command line used.

What glibc and readline versions do you have on your FC5?

Yes I agree, it is more likely to be something else, because piping into mysql works fine. 

I used the Mysql 5.0.21 RPM that is in the updates repository for FC5. When I did it, it was still in the updates-testing repository, but now it's actually moved to the updates one.

http://download.fedora.redhat.com/pub/fedora/linux/core/updates/5/i386/mysql-server-5.0.21...

rpm -qif /usr/lib/libreadline.so.5.0
Version     : 5.0    
Release     : 3.2.1     

rpm -qif /usr/lib/libc.a
Version     : 2.4 
Release     : 8

Please, try to repeat with statically linked generic Client RPM from MySQL, and inform about the results.

hmm, well I will try, but I don't have some spare machines lying around that I can test it on. And I wouldn't want to mess up my workstation to stop me from being able to do things.

I was able to repeat mysql client crash with latest 5.0.23-BK build (on SuSE Linux 9.3). See file uploaded for exact long SQL statement used (downloaded from original reporter's URL).

The following configure options were used:

./configure --prefix=/home/openxs/dbs/5.0 --with-extra-charsets=all \
--with-readline --with-federated-storage-engine \
--with-archive-storage-engine --with-blackhole-storage-engine

Long SELECT that leads mysql command line client to crash

Attachment: 19474.txt (text/plain), 40.73 KiB.

Valgrind points out that a write outside an array occurs at line 763 of display.c, it's in the CHECK_LPOS macro. If that macro is expanded and recompiled it will occur on this line:
_rl_wrapped_line[newlines] = _rl_wrapped_multicolumn;

Also tested to paste the same large query in bash, it will not crash/segfault but will become totally unresponsive.

Hmm, did I say bash didn't crash from this one? That is not true. Will try to file a bugreport upstream.

A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/18939

ChangeSet@1.2392, 2007-01-29 12:24:08+01:00, msvensson@pilot.mysql.com +1 -0
  Bug#19474 readline bug: mysql: free(): invalid pointer
   - Write to uninitialised memory occured since _rl_rapped_lines buffer
     was not extended in CHECK_INV_LBREAKS macro
   - Patch submitted to bug-readline@gnu.org

Available in 5.0.36 and 5.1.16-beta.

Noted in 5.0.36, 5.1.16 changelogs.

The readline library wrote to uninitialized memory, causing mysql to
crash.