Bug #57796 mysqld segfaults, 'alter table' featured in backtrace
Submitted: 28 Oct 2010 9:47 Modified: 24 Jul 2015 18:45
Reporter: Matthew Bloch Email Updates:
Status: No Feedback Impact on me:
None 
Category:MySQL Server: GIS Severity:S1 (Critical)
Version:5.1.51-linux-x86_64-glibc23 OS:Linux
Assigned to: CPU Architecture:Any
Tags: crash

[28 Oct 2010 9:47] Matthew Bloch
Description:
After running for about 1 week on the community build of 5.1.51, a customer's busy application triggered a segfault in mysqld.

From the backtrace, the customer may have been running an "alter table" command at the time - I'm not sure whether this was something administrative, or a normal part of the customer's application.  

We log the output of "show processlist" every minute on the live database servers which might give you a clue what was going on - will send the last one before it crashed as a private comment.

Here is the backtrace:

key_buffer_size=536870912
read_buffer_size=8388608
max_used_connections=385
max_threads=500
threads_connected=31
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 5649463 K
bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

thd: 0x7f0ba51c4e20
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0x45902110 thread_stack 0x40000
./bin/mysqld(my_print_stacktrace+0x2e)[0x8b413e]
./bin/mysqld(handle_segfault+0x322)[0x5e36a2]
/lib/libpthread.so.0[0x7f0d55f24a80]
./bin/mysqld(rtree_split_page+0x425)[0x894ae5]
./bin/mysqld(rtree_add_key+0x101)[0x88e3f1]
./bin/mysqld[0x88ccdf]
./bin/mysqld[0x88cec3]
./bin/mysqld[0x88cec3]
./bin/mysqld[0x88cec3]
./bin/mysqld[0x88cec3]
./bin/mysqld[0x88d110]
./bin/mysqld(rtree_insert+0x23)[0x88d723]
./bin/mysqld(mi_write+0x451)[0x878e31]
./bin/mysqld(_ZN7handler12ha_write_rowEPh+0x67)[0x6d78a7]
./bin/mysqld(_Z17mysql_alter_tableP3THDPcS1_P24st_ha_create_informationP10TABLE_LISTP10Alter_infojP8st_orderb+0x2f09)[0x6f1119]
./bin/mysqld(_Z21mysql_execute_commandP3THD+0xf1a)[0x5f4bda]
./bin/mysqld(_Z11mysql_parseP3THDPcjPPKc+0x3d8)[0x5f9848]
./bin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcj+0x542)[0x5f9d92]
./bin/mysqld(_Z10do_commandP3THD+0xe6)[0x5fb096]
./bin/mysqld(handle_one_connection+0x246)[0x5ed6f6]
/lib/libpthread.so.0[0x7f0d55f1cfc7]
/lib/libc.so.6(clone+0x6d)[0x7f0d553bb59d]
Trying to get some variables.
Some pointers may be invalid and cause the dump to abort...
thd->query at 0x7f0be6ae8ce0 is an invalid pointer
thd->thread_id=7342390
thd->killed=NOT_KILLED

The [mysqld] section of /etc/mysql/my.cnf reads like this:

user            = mysql
pid-file        = /var/run/mysqld/mysqld.pid
socket          = /var/run/mysqld/mysqld.sock
port            = 3306
basedir         = /usr/local/mysql-5.1.51-linux-x86_64-glibc23
datadir         = /var/lib/mysql
tmpdir          = /tmp
language        = /usr/local/mysql-5.1.51-linux-x86_64-glibc23/share/english
skip-external-locking
skip-name-resolve
old_passwords   = 1
bind-address            = 0.0.0.0
core-file
skip-locking
key_buffer              = 512M
max_allowed_packet      = 16M
thread_stack            = 256K
thread_cache_size       = 512
myisam-recover         = BACKUP
max_connections        = 500
table_cache            = 2048
thread_concurrency     = 10
query_cache_limit       = 2M
query_cache_size        = 64M
query_cache_type = 1
wait_timeout = 40
sort_buffer_size = 2M
read_buffer_size = 8M
innodb_buffer_pool_size=4G
innodb_log_file_size=5242880
innodb_log_buffer_size=4M
innodb_thread_concurrency=8
innodb_flush_method=O_DIRECT
innodb_flush_log_at_trx_commit=1
log-bin             = /var/log/mysql/mysql-bin.log
log-slave-updates
log-bin-index       = /var/log/mysql/mysql-log.index
relay-log           = /var/log/mysql/relay.log
relay-log-info-file = /var/log/mysql/relay-log.info
relay-log-index     = /var/log/mysql/relay-log.index
master-host = 10.0.2.1
master-user = replication
master-pass = xxx
expire-logs-days        = 4
max_binlog_size         = 100M
sync_binlog=1
binlog_format=ROW

Sorry I don't have the core file as when this happens we had to quickly wipe /var/lib/mysql and get the server moving again as a slave - if it happens again I'll try to remember to fetch it.

The hardware is an HP DL365, 2 x Opteron 2372 HEs, 8GiB RAM.

How to repeat:
Not sure yet, will follow up if it does.

Suggested fix:
None.
[28 Oct 2010 10:02] MySQL Verification Team
Hi Matthew, can you please send output of

SHOW CREATE TABLE shopper_postcode2;
[28 Oct 2010 10:39] Matthew Bloch
Sent privately.  Also NB that was from the current live copy of the database, not the one that crashed - as I said we don't have that data any more as we needed to redeploy quickly.
[16 Nov 2010 21:23] Sveta Smirnova
Thank you for the feedback.

I can not repeat the problem with test data. Looks like more information from your side needed. Please keep core file and table *frm, *MYD and *MYI files if problem occurs next time. Please also check OS error log file in case if it was real memory shortage at crash time.
[17 Dec 2010 0:00] Bugs System
No feedback was provided for this bug for over a month, so it is
being suspended automatically. If you are able to provide the
information that was originally requested, please do so and change
the status of the bug back to "Open".
[24 Jun 2015 18:05] Tianzhou Chen
We also notice the same crash on 5.6.24 on linux

Registers at time of signal: UNKNOWN: 18446744073709551615(0x7fa9b363e680(), 0x7fa9b363e680(), 0x0(), 0x0(), 0x7fa9e3a1cdd8(\310{\301\376@\201\301\376\010\201\301\376\270\200\301\376H\200\301\376\330\177\301\3760\177\301\376@\201\301\376\340~\301\376h~\301\376\270}\301\376\320|\301\376x|\301\376\030|\301\376\350{\301\376O\202\301\376\314\214\301\376\234\214\301\376,\214\301\376\\\213\301\376l\212\301\376\274\210\301\376\314\214\301\376\\\210\301\376|\207\301\376\274\205\301\376\324\203\301\376\014\203\301\376\\\202\301\376\014\202\301\376@\217\301\376X\225\301\376\030\225\301\376\240\224\301\376\360\223\301\3768\223\301\376), 0x7fa9b363f518(\223\024\303\246=\305\275R\223\024\303\246=\305\275R)) at eip 0x7fa9e2638993 (48 8b 55 98 c7 40 08 01) with eax 0 and base pointer 0x7fa9b363eb70

Stacktrace

rtree_split_page(0x7fa9e2638993)
rtree_add_key(0x7fa9e2632fc1)
rtree_insert_req(0x7fa9e262dc2b)
rtree_insert_level(0x7fa9e262dca6)
rtree_insert(0x7fa9e262e314)
mi_write(0x7fa9e2630abb)
handler::ha_write_row(unsigned char*)(0x7fa9e2545160)
write_record(THD*, TABLE*, COPY_INFO*, COPY_INFO*)(0x7fa9e23927c5) mysql_insert(THD*, TABLE_LIST*, List<Item>&, List<List<Item> >&, List<Item>&, List<Item>&, enum_duplicates, bool)(0x7fa9e2395c7d)
mysql_execute_command(THD*)(0x7fa9e236c567)
mysql_parse(THD*, char*, unsigned int, Parser_state*)(0x7fa9e236f6c8) dispatch_command(enum_server_command, THD*, char*, unsigned int)(0x7fa9e2371f27)
do_handle_one_connection(THD*)(0x7fa9e23b2fd6) handle_one_connection(0x7fa9e23b2ff9)
start_thread(0x7fa9e0dcb800) __clone(0x7fa9e08262ed)
[24 Jun 2015 18:45] MySQL Verification Team
Hi, kindly try the newer versions of MySQL.  
I believe it would solve your issue.  If not,  let us know here.
Referring to this in particular (based on stack trace of crash).
----------
Noted in 5.5.44, 5.6.25, 5.7.8, 5.8.0 changelogs.

Loading corrupt spatial data into a MyISAM table could cause the
server to exit during index building.
----------
[25 Jul 2015 1:00] Bugs System
No feedback was provided for this bug for over a month, so it is
being suspended automatically. If you are able to provide the
information that was originally requested, please do so and change
the status of the bug back to "Open".