Bug #62165 Mysql crashes
Submitted: 14 Aug 2011 16:24 Modified: 19 Aug 2011 20:01
Reporter: Deepak S Email Updates:
Status: Verified Impact on me:
None 
Category:MySQL Server: MyISAM storage engine Severity:S2 (Serious)
Version:5.0.92-community-log, 5.6.3 OS:Linux (CentIOS)
Assigned to: CPU Architecture:Any
Tags: crash, glibc
Triage: Needs Triage: D1 (Critical)

[14 Aug 2011 16:24] Deepak S
Description:
Mysql was crashing on the server periodically and upon investigation it was found that the server was crashed while it was working with a Database.

110814 11:22:25 [ERROR] /usr/sbin/mysqld: Incorrect information in file: './db1/phplist_attachment.frm'
110814 11:22:25 [ERROR] /usr/sbin/mysqld: Incorrect information in file: './db1/phplist_bounce.frm'
110814 11:22:25 [ERROR] /usr/sbin/mysqld: Incorrect information in file: './db1/phplist_bounce.frm'
110814 11:22:25 [ERROR] /usr/sbin/mysqld: Incorrect information in file: './db1/phplist_bounceregex.frm'
*** glibc detected *** /usr/sbin/mysqld: double free or corruption (!prev): 0x0a720668 ***
======= Backtrace: =========
/lib/libc.so.6[0x55c6c5]
/lib/libc.so.6(cfree+0x59)[0x55cb09]
/usr/sbin/mysqld(free_root+0x63)[0x83fcd53]
/usr/sbin/mysqld(_Z7openfrmP3THDPKcS2_jjjP8st_table+0x254)[0x81cd9c4]
/usr/sbin/mysqld[0x81c70db]
/usr/sbin/mysqld(_Z10open_tableP3THDP10TABLE_LISTP11st_mem_rootPbj+0x704)[0x81c8774]
/usr/sbin/mysqld(_Z11open_tablesP3THDPP10TABLE_LISTPjj+0x2e0)[0x81c8fe0]
/usr/sbin/mysqld(_Z30open_normal_and_derived_tablesP3THDP10TABLE_LISTj+0x27)[0x81c91c7]
/usr/sbin/mysqld(_Z18mysqld_list_fieldsP3THDP10TABLE_LISTPKc+0x23)[0x826b863]
/usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcj+0x17ac)[0x81a24fc]
/usr/sbin/mysqld(handle_one_connection+0x9bf)[0x81a31ff]
/lib/libpthread.so.0[0x65a832]
/lib/libc.so.6(clone+0x5e)[0x5c546e]
[ rest will be attached ]

System memory didn't seem to get full at the time of crashing though.

I was then able to reproduce the issue by just running 'use db1';

CentOS 5
glibc version 2.5-58

How to repeat:
run 'use db1';
[14 Aug 2011 16:26] Deepak S
More log at the time of crash

Attachment: crash.txt (text/plain), 12.15 KiB.

[18 Aug 2011 19:05] Sveta Smirnova
Thank you for the report.

How many tables do you have in database db1? Is it possible to send us in hidden comment this database directory compressed?
[18 Aug 2011 21:04] Deepak S
Attached mysql db file you have requested.

Attachment: plst1.tar.gz (application/x-gzip, text), 110.06 KiB.

[19 Aug 2011 20:01] Sveta Smirnova
Thank you for the datadir.

Verified as described.

Backtrace from version 5.6.3:

Thread pointer: 0xc28d290
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0x4a0930c8 thread_stack 0x20000
./bin/mysqld(my_print_stacktrace+0x35)[0x94a38e]
./bin/mysqld(handle_segfault+0x31f)[0x562714]
/lib64/libpthread.so.0[0x3429e0dd40]
./bin/mysqld(_db_enter_+0xd4)[0xc011bb]
./bin/mysqld(alloc_root+0x37)[0x93c43e]
./bin/mysqld(strmake_root+0x2b)[0x93ce22]
./bin/mysqld[0x7bc8a9]
./bin/mysqld(_Z14open_table_defP3THDP11TABLE_SHAREj+0x49e)[0x7bb6b4]
./bin/mysqld(_Z15get_table_shareP3THDP10TABLE_LISTPKcjjPij+0x17e)[0x68e756]
./bin/mysqld[0x68eaa4]
./bin/mysqld(_Z10open_tableP3THDP10TABLE_LISTP11st_mem_rootP18Open_table_context+0x805)[0x693361]
./bin/mysqld[0x695e5a]
./bin/mysqld(_Z11open_tablesP3THDPP10TABLE_LISTPjjP19Prelocking_strategy+0x34f)[0x696d01]
./bin/mysqld(_Z30open_normal_and_derived_tablesP3THDP10TABLE_LISTj+0xce)[0x697dc5]
./bin/mysqld(_Z18mysqld_list_fieldsP3THDP10TABLE_LISTPKc+0x79)[0x757546]
./bin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcj+0x1384)[0x6e9260]
./bin/mysqld(_Z10do_commandP3THD+0x298)[0x6e7cc9]
./bin/mysqld(_Z24do_handle_one_connectionP3THD+0x199)[0x6bf33e]
./bin/mysqld(handle_one_connection+0x33)[0x6bee48]
/lib64/libpthread.so.0[0x3429e061b5]
/lib64/libc.so.6(clone+0x6d)[0x34292cd39d]

As workaround I recommend you to run myisamchk with repair option on tables in this database one-by-one, then test.
[19 Aug 2011 20:12] Sveta Smirnova
Cause seems to be phplist_bounce, phplist_bounceregex and phplist_config tables:

myisamchk: error: 'phplist_bounce.MYI' is not a MyISAM-table
- recovering (with keycache) MyISAM-table 'phplist_bounceregex_bounce.MYI'
Data records: 0
myisamchk: error: 'phplist_bounceregex.MYI' is not a MyISAM-table
myisamchk: error: 'phplist_config.MYI' is not a MyISAM-table
[11 Dec 2011 20:22] M Chojnowski
turn off query_cache_strip_comments
Related to https://bugs.launchpad.net/percona-server/+bug/705688