Bug #15731 My SQL 5 restarts
Submitted: 14 Dec 2005 7:52 Modified: 21 Jan 2006 11:25
Reporter: Syed Kazim Hussain Email Updates:
Status: No Feedback Impact on me:
None 
Category:MySQL Server Severity:S1 (Critical)
Version:5.0.16-standard-log OS:Linux (FC3 Linux)
Assigned to: CPU Architecture:Any

[14 Dec 2005 7:52] Syed Kazim Hussain
Description:
My SQL 5 restarts giving signal 12 although there was memory available. The log file shows that My SQL restarts on this query which does not look that expensive and proper indexes are created :

select distinct languageId from  productlanguage  where productid=11111668 and isactive=1

When we ran the resolve_stack_trace utlity, it gave the following stack trace:

[root@backup1 tmp]# resolve_stack_dump -s /tmp/mysqld.sym -n  mysqld.stack
0x8150650 handle_segfault + 356
0x59a420 (?)
0x81edff2 _ZN7ha_heap9write_rowEPc + 66
0x81edff2 _ZN7ha_heap9write_rowEPc + 66
0x819c9b3 _Z9end_writeP4JOINP13st_join_tableb + 227
0x819be75 _Z20evaluate_join_recordP4JOINP13st_join_tableiPc + 329
0x8195cda _Z10sub_selectP4JOINP13st_join_tableb + 270
0x819bb28 _Z9do_selectP4JOINP4ListI4ItemEP8st_tableP9Procedure + 268
0x81913f9 _ZN4JOIN4execEv + 1309
0x81929f5  _Z12mysql_selectP3THDPPP4ItemP13st_table_listjR4ListIS1_ES2_jP8st_orde rSB_S2_SB_mP13select_resultP18st_select_lex_unitP13st_sel + 305
0x818ebd2 _Z13handle_selectP3THDP6st_lexP13select_resultm + 234
0x8164175 _Z21mysql_execute_commandP3THD + 601
0x81b146a _ZN18Prepared_statement7executeEP6Stringb + 634
0x81b0002 _Z18mysql_stmt_executeP3THDPcj + 302
0x816393a _Z16dispatch_command19enum_server_commandP3THDPcj + 4914
0x81625cd _Z10do_commandP3THD + 129
0x8161aad handle_one_connection + 569
0x665341 (?)
0x2076fe (?) 

Here is the error-log file 

051212 23:16:26 [ERROR] /usr/local/mysql/bin/mysqld: Got error 12 from storage engine
051212 23:16:26 [ERROR] /usr/local/mysql/bin/mysqld: Sort aborted
mysqld got signal 11;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help diagnose
the problem, but since we have already crashed, something is definitely wrong
and this may fail.

key_buffer_size=469762048
read_buffer_size=2093056
max_used_connections=114
max_connections=200
threads_connected=69
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_connections = 1277150 K
bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

thd=0x53a29ba0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
Cannot determine thread, fp=0x8fd7b9c, backtrace may not be correct.
Stack range sanity check OK, backtrace follows:
0x8150650
0x59a420
0x81edff2
0x81edff2
0x819c9b3
0x819be75
0x8195cda
0x819bb28
0x81913f9
0x81929f5
0x818ebd2
0x8164175
0x81b146a
0x81b0002
0x816393a
0x81625cd
0x8161aad
0x665341
0x2076fe
New value of fp=(nil) failed sanity check, terminating stack trace!
Please read http://dev.mysql.com/doc/mysql/en/Using_stack_trace.html and follow instructions on how to resolve the stack trace. Resolved
stack trace is much more helpful in diagnosing the problem, so please do
resolve it
Trying to get some variables.
Some pointers may be invalid and cause the dump to abort...
thd->query at 0xa9b0710 = select distinct languageId from  productlanguage  where productid=11111668 and isactive=1
thd->thread_id=897
The manual page at http://www.mysql.com/doc/en/Crashing.html contains
information that should help you find out what is causing the crash

Any help would be appreciated.

How to repeat:
Unable to reproduce in a staging environment
[14 Dec 2005 11:18] Valeriy Kravchuk
Thank you for a problem report. Please, send the results of the SHOW CREATE TABLE productlanguage statement, inform about the number of rows in this table, send the content of your my.cnf file and describe your hardware (how much RAM does it really have).
[14 Dec 2005 11:31] Syed Kazim Hussain
We have hosted mysql on a dedicated server machine with 4 GB RAM on x86 platform.

show create Table productlanguage :-
CREATE TABLE `productlanguage` (
  `productlanguageid` int(10) unsigned NOT NULL auto_increment,
  `productid` int(10) unsigned NOT NULL default '0',
  `languageid` smallint(5) unsigned NOT NULL default '0',
  `isactive` tinyint(1) NOT NULL default '0',
  `createdate` datetime NOT NULL default '0000-00-00 00:00:00',
  `updatedate` datetime NOT NULL default '0000-00-00 00:00:00',
  `createbyid` int(10) unsigned NOT NULL default '0',
  `updatebyid` int(10) unsigned NOT NULL default '0',
  `productstatusid` tinyint(4) NOT NULL default '0',
  `ispartiallyfilled` tinyint(1) NOT NULL default '0',
  PRIMARY KEY  (`productlanguageid`),
  KEY `productid` (`productid`),
  KEY `updatebyid` (`updatebyid`),
  KEY `createbyid` (`createbyid`),
  KEY `languageid` (`languageid`),
  KEY `productstatusid` (`productstatusid`),
  KEY `ispartiallyfilled` (`ispartiallyfilled`),
  KEY `productid_2` (`productid`,`productstatusid`),
  CONSTRAINT `fk_plps` FOREIGN KEY (`productstatusid`) REFERENCES `productstatuslookup` (`productstatusid`)
) ENGINE=InnoDB DEFAULT CHARSET=latin1

Number of rows in productlanguage :-  1497793

My.cnf file contents :- 
# Example MySQL config file for medium systems.
#
# This is for a system with little memory (32M - 64M) where MySQL plays
# an important part, or systems up to 128M where MySQL is used together with
# other programs (such as a web server)
#
# You can copy this file to
# /etc/my.cnf to set global options,
# mysql-data-dir/my.cnf to set server-specific options (in this
# installation this directory is /usr/local/mysql/data) or
# ~/.my.cnf to set user-specific options.
#
# In this file, you can use all long options that a program supports.
# If you want to know which options a program supports, run the program
# with the "--help" option.

# The following options will be passed to all MySQL clients
[client]
#password	= your_password
port		= 3306
socket		= /tmp/mysql.sock

# Here follows entries for some specific programs

# The MySQL server
[mysqld]
port		= 3306
socket		= /tmp/mysql.sock
skip-locking
key_buffer = 512M
max_allowed_packet = 256M
table_cache = 512000
query_cache_size = 256M
sort_buffer_size = 2M
net_buffer_length = 8K
read_buffer_size = 2M
read_rnd_buffer_size = 8M
myisam_sort_buffer_size = 64M  
set-variable = lower_case_table_names=1
datadir=/export/disk0/data

# Don't listen on a TCP/IP port at all. This can be a security enhancement,
# if all processes that need to connect to mysqld run on the same host.
# All interaction with mysqld must be made via Unix sockets or named pipes.
# Note that using this option without enabling named pipes on Windows
# (via the "enable-named-pipe" option) will render mysqld useless!
# 
#skip-networking
log-slow-queries = /export/disk0/mysql/mysql-slow.log
set-variable = max_connections=250

# Replication Master Server (default)
# binary logging is required for replication
#log-bin=mysql-bin

# required unique id between 1 and 2^32 - 1
# defaults to 1 if master-host is not set
# but will not function as a master if omitted
server-id	= 1

# Replication Slave (comment out master section to use this)
#
# To configure this host as a replication slave, you can choose between
# two methods :
#
# 1) Use the CHANGE MASTER TO command (fully described in our manual) -
#    the syntax is:
#
#    CHANGE MASTER TO MASTER_HOST=<host>, MASTER_PORT=<port>,
#    MASTER_USER=<user>, MASTER_PASSWORD=<password> ;
#
#    where you replace <host>, <user>, <password> by quoted strings and
#    <port> by the master's port number (3306 by default).
#
#    Example:
#
#    CHANGE MASTER TO MASTER_HOST='125.564.12.1', MASTER_PORT=3306,
#    MASTER_USER='joe', MASTER_PASSWORD='secret';
#
# OR
#
# 2) Set the variables below. However, in case you choose this method, then
#    start replication for the first time (even unsuccessfully, for example
#    if you mistyped the password in master-password and the slave fails to
#    connect), the slave will create a master.info file, and any later
#    change in this file to the variables' values below will be ignored and
#    overridden by the content of the master.info file, unless you shutdown
#    the slave server, delete master.info and restart the slaver server.
#    For that reason, you may want to leave the lines below untouched
#    (commented) and instead use CHANGE MASTER TO (see above)
#
# required unique id between 2 and 2^32 - 1
# (and different from the master)
# defaults to 2 if master-host is set
# but will not function as a slave if omitted
#server-id       = 2
#
# The replication master for this slave - required
#master-host     =   <hostname>
#
# The username the slave will use for authentication when connecting
# to the master - required
#master-user     =   <username>
#
# The password the slave will authenticate with when connecting to
# the master - required
#master-password =   <password>
#
# The port the master is listening on.
# optional - defaults to 3306
#master-port     =  <port>
#
# binary logging - not required for slaves, but recommended
#log-bin=mysql-bin

# Point the following paths to different dedicated disks
tmpdir		= /tmp/		
#log-update 	= /path-to-dedicated-directory/hostname
# Uncomment the following if you are using BDB tables
#bdb_cache_size = 4M
#bdb_max_lock = 10000

# Uncomment the following if you are using InnoDB tables
innodb_data_home_dir = /export/disk0/data/
innodb_data_file_path = ibdata1:2000M;ibdata2:2000M;ibdata3:2000M;ibdata4:2000M;ibdata5:2000M;ibdata6:2000M;ibdata7:2000M;ibdata8:2000M;ibdata9:2000M;ibdata10:2000M;ibdata11:2000M;ibdata12:2000M;ibdata13:2000M;ibdata14:2000M;ibdata15:2000M:autoextend
innodb_log_group_home_dir = /export/disk0/data/
innodb_log_arch_dir = /export/disk0/data/
set-variable = innodb_log_files_in_group=2
# You can set .._buffer_pool_size up to 50 - 80 %
# of RAM but beware of setting memory usage too high

innodb_buffer_pool_size = 712M
innodb_additional_mem_pool_size = 20M
#Set .._log_file_size to 25 % of buffer pool size
innodb_log_file_size = 128M
innodb_log_buffer_size = 8M
innodb_flush_log_at_trx_commit = 1
innodb_lock_wait_timeout = 50

[mysqldump]
quick
max_allowed_packet = 256M

[mysql]
no-auto-rehash
# Remove the next comment character if you are not familiar with SQL
#safe-updates

[isamchk]
key_buffer = 256M
sort_buffer_size = 256M
read_buffer = 2M
write_buffer = 2M

[myisamchk]
key_buffer = 256M
sort_buffer_size = 256M
read_buffer = 2M
write_buffer = 2M

[mysqlhotcopy]
interactive-timeout
[14 Dec 2005 11:35] Syed Kazim Hussain
Sorry, discard the last my.cnf file. Here is the one we are using :

# Example MySQL config file for medium systems.
#
# This is for a system with little memory (32M - 64M) where MySQL plays
# an important part, or systems up to 128M where MySQL is used together with
# other programs (such as a web server)
#
# You can copy this file to
# /etc/my.cnf to set global options,
# mysql-data-dir/my.cnf to set server-specific options (in this
# installation this directory is /usr/local/mysql/data) or
# ~/.my.cnf to set user-specific options.
#
# In this file, you can use all long options that a program supports.
# If you want to know which options a program supports, run the program
# with the "--help" option.

# The following options will be passed to all MySQL clients
[client]
#password       = your_password
port            = 3306
socket          = /tmp/mysql.sock

# Here follows entries for some specific programs

# The MySQL server
[mysqld]
port            = 3306
socket          = /tmp/mysql.sock
skip-locking
key_buffer_size = 448M
max_allowed_packet = 512M
table_cache = 64
query_cache_type = 1 
query_cache_size = 256M
#query_cache_min_res_unit = 8K
sort_buffer_size = 2M
net_buffer_length = 8K
read_buffer_size = 2M
read_rnd_buffer_size = 10M
myisam_sort_buffer_size = 64M  
set-variable = lower_case_table_names=1
datadir=/export/disk0/data
binlog-do-db=cms
binlog-do-db=MySQL_Admin
#binlog-do-db=oms

# Don't listen on a TCP/IP port at all. This can be a security enhancement,
# if all processes that need to connect to mysqld run on the same host.
# All interaction with mysqld must be made via Unix sockets or named pipes.
# Note that using this option without enabling named pipes on Windows
# (via the "enable-named-pipe" option) will render mysqld useless!
# 
#skip-networking
log-slow-queries = /export/disk0/data/mysql-slow.log
set-variable = max_connections=200

# Replication Master Server (default)
# binary logging is required for replication
log-bin=mysql-bin
log
# required unique id between 1 and 2^32 - 1
# defaults to 1 if master-host is not set
# but will not function as a master if omitted
server-id       = 1

# Replication Slave (comment out master section to use this)
#
# To configure this host as a replication slave, you can choose between
# two methods :
#
# 1) Use the CHANGE MASTER TO command (fully described in our manual) -
#    the syntax is:
#
#    CHANGE MASTER TO MASTER_HOST=<host>, MASTER_PORT=<port>,
#    MASTER_USER=<user>, MASTER_PASSWORD=<password> ;
#
#    where you replace <host>, <user>, <password> by quoted strings and
#    <port> by the master's port number (3306 by default).
#
#    Example:
#
#    CHANGE MASTER TO MASTER_HOST='125.564.12.1', MASTER_PORT=3306,
#    MASTER_USER='joe', MASTER_PASSWORD='secret';
#
# OR
#
# 2) Set the variables below. However, in case you choose this method, then
#    start replication for the first time (even unsuccessfully, for example
#    if you mistyped the password in master-password and the slave fails to
#    connect), the slave will create a master.info file, and any later
#    change in this file to the variables' values below will be ignored and
#    overridden by the content of the master.info file, unless you shutdown
#    the slave server, delete master.info and restart the slaver server.
#    For that reason, you may want to leave the lines below untouched
#    (commented) and instead use CHANGE MASTER TO (see above)
#
# required unique id between 2 and 2^32 - 1
# (and different from the master)
# defaults to 2 if master-host is set
# but will not function as a slave if omitted
#server-id       = 2
#
# The replication master for this slave - required
#master-host     =   <hostname>
#
# The username the slave will use for authentication when connecting
# to the master - required
#master-user     =   <username>
#
# The password the slave will authenticate with when connecting to
# the master - required
#master-password =   <password>
#
# The port the master is listening on.
# optional - defaults to 3306
#master-port     =  <port>
#
# binary logging - not required for slaves, but recommended
#log-bin=mysql-bin

# Point the following paths to different dedicated disks
tmpdir          = /tmp/
#log-update     = /path-to-dedicated-directory/hostname
# Uncomment the following if you are using BDB tables
#bdb_cache_size = 4M
#bdb_max_lock = 10000

# Uncomment the following if you are using InnoDB tables
innodb_data_home_dir = /export/disk0/data/
innodb_data_file_path = ibdata1:2000M;ibdata2:2000M;ibdata3:2000M;ibdata4:2000M;ibdata5:2000M;ibdata6:2000M;ibdata7:2000M;ibdata8:2000M;ibdata
9:2000M;ibdata10:2000M;ibdata11:2000M;ibdata12:2000M;ibdata13:2000M;ibdata14:2000M;ibdata15:2000M:autoextend
innodb_log_group_home_dir = /export/disk0/data/
innodb_log_arch_dir = /export/disk0/data/
set-variable = innodb_log_files_in_group=2
# You can set .._buffer_pool_size up to 50 - 80 %
# of RAM but beware of setting memory usage too high

innodb_buffer_pool_size = 1280M
innodb_additional_mem_pool_size = 20M
#Set .._log_file_size to 25 % of buffer pool size
innodb_log_file_size = 128M
innodb_log_buffer_size = 8M
innodb_flush_log_at_trx_commit = 1
innodb_lock_wait_timeout = 50

[mysqldump]
quick
max_allowed_packet = 512M

[mysql]
no-auto-rehash
# Remove the next comment character if you are not familiar with SQL
#safe-updates

[isamchk]
key_buffer = 256M
sort_buffer_size = 256M
read_buffer = 2M
write_buffer = 2M

[myisamchk]
key_buffer = 256M
sort_buffer_size = 256M
read_buffer = 2M
write_buffer = 2M

[mysqlhotcopy]
interactive-timeout
[14 Dec 2005 11:49] Aleksey Kishkin
..'got error 12 from ..'
perror 12 - not enough space

Could you check if you have space left on device that you use for data?
[14 Dec 2005 11:55] Syed Kazim Hussain
400 GB space left on device for data
[14 Dec 2005 12:06] Aleksey Kishkin
what about /tmp (that uses for temporary files)?
[14 Dec 2005 12:18] Syed Kazim Hussain
2 GB
[21 Dec 2005 11:25] Aleksey Kishkin
Syed, for fixing we need a reproduceable testcase. Could you suggest us how to reproduce this bug?
[22 Jan 2006 0:00] Bugs System
No feedback was provided for this bug for over a month, so it is
being suspended automatically. If you are able to provide the
information that was originally requested, please do so and change
the status of the bug back to "Open".