Bug #57943 Constant server crashes
Submitted: 3 Nov 2010 6:46 Modified: 9 Mar 2011 15:29
Reporter: Oliver DSouza Email Updates:
Status: No Feedback Impact on me:
None 
Category:MySQL Server: General Severity:S2 (Serious)
Version:5.0.77 OS:Other (CentOS)
Assigned to: CPU Architecture:Any
Tags: free(), malloc, Signal 11, signal 6

[3 Nov 2010 6:46] Oliver DSouza
Description:
I have a MySQL Server version: 5.0.77-log

Which seems to be crashing constantly every few mins, 
The reason for crashing is any thing from Signal 11, Signal 6, Assertion Failure, to memory corruption eg Malloc failed or Free ()...

This server is on Dual Qaud Core, with 8 GB Ram
Running on CentOS 5.x

FYI A lot of innodb tables are used. 
Overall DB size is 6-7 GB

Not really sure whats causing these constant crashes.
-------------------------
Extract of mysqld.log
-------------------------
101101 12:20:01 - mysqld got signal 11 ;
....
....
101101 12:29:01 - mysqld got signal 11 ;
..... //May be this was because i set force recovery to 1
InnoDB: A new raw disk partition was initialized or
InnoDB: innodb_force_recovery is on: we do not allow
InnoDB: database modifications by the user. Shut down
InnoDB: mysqld and edit my.cnf so that newraw is replaced
InnoDB: with raw, and innodb_force_... is removed.
..................
InnoDB: database modifications by the user. Shut down
InnoDB: mysqld and edit my.cnf so that newraw is replaced
InnoDB: with raw, and innodb_force_... is removed.
101101 14:50:11 [Note] /usr/libexec/mysqld: Normal shutdown
...........
....
101101 15:24:04  InnoDB: Error: MySQL is freeing a thd
InnoDB: though trx->n_mysql_tables_in_use is 1
InnoDB: and trx->mysql_n_tables_locked is 124.
TRANSACTION 0 598822756, not started, process no 28938, OS thread id 1629404048
mysql tables in use 1, locked 124
MySQL thread id 987, query id 43042622 localhost mv_php_usr
 len 860; hex 065...   Thank you!te is to cater to the ne;101102  9:13:47 - mysqld got signal 11 ;
......
....
101102  9:46:29 - mysqld got signal 11 ;
...
101102 10:08:01InnoDB: Assertion failure in thread 1629285264 in file row0mysql.c line 145
InnoDB: Failing assertion: lenlen == 1
.....
.....
*** glibc detected *** /usr/libexec/mysqld: free(): invalid pointer: 0x6080eb50 ***
======= Backtrace: =========
/lib/libc.so.6[0x28e0f1]
/lib/libc.so.6(cfree+0x90)[0x291bc0]
/usr/libexec/mysqld(delete_dynamic+0x18)[0x848fe08]
/usr/libexec/mysqld(_ZN3THD7cleanupEv+0x8d)[0x817e04d]
/usr/libexec/mysqld(_Z10end_threadP3THDb+0x16)[0x818bc86]
/usr/libexec/mysqld(handle_one_connection+0x403)[0x81ad1b3]
/lib/libpthread.so.0[0x39f49b]
/lib/libc.so.6(clone+0x5e)[0x2f642e]
======= Memory map: ========
00169000-0016a000 r-xp 00169000 00:00 0          [vdso]
00179000-00182000 r-xp 00000000 fd:00 53478649   /lib/libcrypt-2.5.so
00182000-00183000 r--p 00008000 fd:00 53478649   /lib/libcrypt-2.5.so
00183000-00184000 rw-p 00009000 fd:00 53478649   /lib/libcrypt-2.5.so
00184000-001ab000 rw-p 00184000 00:00 0 
...
...
..
101102 12:03:44 - mysqld got signal 11 ;
....
...
*** glibc detected *** /usr/libexec/mysqld: malloc(): memory corruption: 0x60d23ec8 ***
======= Backtrace: =========
/lib/libc.so.6[0x28e883]
/lib/libc.so.6(__libc_malloc+0x7b)[0x2903ab]
/usr/libexec/mysqld(my_malloc+0x25)[0x8489355]
/usr/libexec/mysqld(init_alloc_root+0x73)[0x8489ce3]
/usr/libexec/mysqld(_Z14init_sql_allocP11st_mem_rootjj+0x20)[0x8158520]
.....
....
*** glibc detected *** /usr/libexec/mysqld: free(): invalid next size (fast): 0x0b2eea20 ***
======= Backtrace: =========
/lib/libc.so.6[0x28e0f1]
/lib/libc.so.6(cfree+0x90)[0x291bc0]
/usr/libexec/mysqld(_Z19close_thread_tablesP3THDbb+0x55)[0x81cee95]
/usr/libexec/mysqld(_ZN18Prepared_statement12cleanup_stmtEv+0x49)[0x8202c39]
/usr/libexec/mysqld(_ZN18Prepared_statement7prepareEPKcj+0x1c8)[0x8202e98]
/usr/libexec/mysqld(_Z18mysql_stmt_prepareP3THDPKcj+0xcb)[0x8204e2b]
/usr/libexec/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcj+0x1069)[0x81ac439]
/usr/libexec/mysqld(handle_one_connection+0xa78)[0x81ad828]
/lib/libpthread.so.0[0x39f49b]
/lib/libc.so.6(clone+0x5e)[0x2f642e]
======= Memory map: ========
00179000-00182000 r-xp 00000000 fd:00 53478649   /lib/libcrypt-2.5.so
00182000-00183000 r--p 00008000 fd:00 53478649   /lib/libcrypt-2.5.so
00183000-00184000 rw-p 00009000 fd:00 53478649   /lib/libcrypt-2.5.so
00184000-001ab000 rw-p 00184000 00:00 0 
....
...
101102 13:17:11 - mysqld got signal 11 ;
....
....
...
101102 15:11:33  InnoDB: Error: MySQL is freeing a thd
InnoDB: though trx->n_mysql_tables_in_use is 1
InnoDB: and trx->mysql_n_tables_locked is 7670.
TRANSACTION 0 602727896, not started, process no 2261, OS thread id 1629776784
mysql tables in use 1, locked 7670
MySQL thread id 5309, query id 113616026 localhost mv_php_usr
 len 860; hex ....

*** glibc detected *** /usr/libexec/mysqld: free(): invalid pointer: 0x60d1cc18 ***
======= Backtrace: =========
/lib/libc.so.6[0x28e0f1]
/lib/libc.so.6(cfree+0x90)[0x291bc0]
/usr/libexec/mysqld[0x8212d86]
/usr/libexec/mysqld(_Z19ha_close_connectionP3THD+0x2b)[0x824b87b]
/usr/libexec/mysqld(_ZN3THDD0Ev+0x80)[0x81812f0]
/usr/libexec/mysqld(_Z10end_threadP3THDb+0x3e)[0x818bcae]
/usr/libexec/mysqld(handle_one_connection+0x403)[0x81ad1b3]
/lib/libpthread.so.0[0x39f49b]
/lib/libc.so.6(clone+0x5e)[0x2f642e]
======= Memory map: ========
00110000-00119000 r-xp 00000000 fd:00 53477416   /lib/libnss_files-2.5.so
00119000-0011a000 r--p 00008000 fd:00 53477416   /lib/libnss_files-2.5.so
0011a000-0011b000 rw-p 00009000 fd:00 53477416   /lib/libnss_files-2.5.so
00179000-00182000 r-xp 00000000 fd:00 53478649   /lib/libcrypt-2.5.so
00182000-00183000 r--p 00008000 fd:00 53478649   /lib/libcrypt-2.5.so
00183000-00184000 rw-p 00009000 fd:00 53478649   /lib/libcrypt-2.5.so
00184000-001ab000 rw-p 00184000 00:00 0 

-----------------------
my.cnf 
-----------------------
[mysqld]
datadir=/var/lib/mysql
socket=/var/lib/mysql/mysql.sock
user=mysql
skip_name_resolve
#bind-address=192.168.1.3

#General Querieis
log=/var/log/mysql_log_queries.log
#Slow Queries
log_slow_queries=/var/log/mysql_slow_queries.log
long_query_time=2
# Default to using old password format for compatibility with mysql 3.x
# clients (those using the mysqlclient10 compatibility package).
old_passwords=1

# Disabling symbolic-links is recommended to prevent assorted security risks;
# to do so, uncomment this line:
# symbolic-links=0

#MyIsam Variables
myisam_sort_buffer_size=64M
#MySql Variables
key_buffer_size=128M
max_allowed_packet=1M
table_cache=1024
sort_buffer_size=1M
read_buffer_size=1M
read_rnd_buffer_size=4M
thread_cache=16
flush
delay_key_write=Off
#thread_concuency=4

#InnoDB Settings
innodb_buffer_pool_size = 1G
innodb_additional_mem_pool_size = 64M
innodb_log_file_size=256M
innodb_log_buffer_size = 32M
innodb_flush_log_at_trx_commit = 1
innodb_thread_concurrency=8
innodb_file_per_table
#innodb_force_recovery=1

#Query Cache
query_cache_size=128M
query_cache_type=1
#query_cache_size=0
#query_cache_type=0

max_connections=300

#Replication
log-bin=/var/log/mysql/mysql-bin.log
binlog-do-db=mv_sys_data
server-id=1
sync_binlog=1

[mysqldump]
quick
max_allowed_packet=16M

[mysqld_safe]
log-error=/var/log/mysqld.log
pid-file=/var/run/mysqld/mysqld.pid

How to repeat:
May be its specific to my hardware or config or something..
Just start the server and let it work for some time and it will restart.
[3 Nov 2010 8:12] Valeriy Kravchuk
Please, upload the entire error log (compressed if it is big) to the bug report.
[9 Nov 2010 5:55] Oliver DSouza
Mysqld Compressed file uploaded.
[11 Nov 2010 10:23] Oliver DSouza
Guys, wondering if there is any update?
[24 Nov 2010 13:12] Susanne Ebrecht
Hello Oliver,

please look at the end to the error log and look into the row which start with this:

101103 10:01:39 [Warning] ...

Sounds user and host familiar to me or is it hack?
[25 Nov 2010 7:46] Oliver DSouza
Sorry Susanne, i did not get what you said, 
Can you please explain?
[10 Dec 2010 7:30] Oliver DSouza
Attached Back Track for Signal 11

Attachment: mysqld.stack.resolved (application/octet-stream, text), 8.85 KiB.

[16 Dec 2010 21:56] Sveta Smirnova
Thank you for the feedback.

Please try with current version 5.0.91 and if problem still exists send output of SHOW CREATE TABLE stock_quotes and SHOW TABLE STATUS LIKE 'stock_quotes'
[27 Dec 2010 18:37] Oliver DSouza
CREATE TABLE `stock_quotes` (
 `market_id` int(11) NOT NULL,
 `symbol` varchar(20) NOT NULL,
 `series` char(2) NOT NULL,
 `date` date NOT NULL,
 `time` time NOT NULL,
 `last_traded_price` decimal(10,2) NOT NULL,
 `volume` bigint(20) default NULL,
 `net_change` decimal(10,2) NOT NULL,
 `open_price` decimal(10,2) NOT NULL,
 `close_price` decimal(10,2) NOT NULL,
 `interim_high` decimal(10,2) default NULL,
 `interim_low` decimal(10,2) default NULL,
 `dt` datetime default NULL,
 PRIMARY KEY  (`symbol`,`date`,`time`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8
[27 Dec 2010 18:40] Oliver DSouza
Name	Engine	Version	Row_format	Rows	Avg_row_length	Data_length	Max_data_length	Index_length	Data_free	Auto_increment	Create_time	Update_time	Check_time	Collation	Checksum	Create_options	Comment
stock_quotes	InnoDB	10	Compact	42971295	120	5194645504	0	0	0	NULL	2010-12-10 17:27:18	NULL	NULL	utf8_general_ci	NULL	 	InnoDB free: 106496 kB
[28 Dec 2010 17:38] MySQL Verification Team
Have you tested version 5.0.91 as Sveta asked you?. Thanks in advance.
[10 Mar 2011 0:00] Bugs System
No feedback was provided for this bug for over a month, so it is
being suspended automatically. If you are able to provide the
information that was originally requested, please do so and change
the status of the bug back to "Open".