MySQL Bugs: #17958: MySQL Server Crash

Bug #17958	MySQL Server Crash
Submitted:	6 Mar 2006 11:02	Modified:	6 May 2006 14:18
Reporter:	Wachana Saikaewtet	Email Updates:
Status:	No Feedback	Impact on me:	None
Category:	MySQL Server	Severity:	S2 (Serious)
Version:	4.0.26-Max-Log	OS:	Linux (Linux RH9)
Assigned to:		CPU Architecture:	Any

Description:
MySQL server is Crash.

This is mysql error log file.
----------------------------------------------------------------------------------------------
/usr/sbin/mysqld-max: ready for connections.
Version: '4.0.26-Max-log' socket: '/var/lib/mysql/mysql.sock' port: 3306 Official MySQL RPM
mysqld got signal 11;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help diagnose
the problem, but since we have already crashed, something is definitely wrong
and this may fail.

key_buffer_size=209715200
read_buffer_size=1044480
max_used_connections=119
max_connections=300
threads_connected=120
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_connections = 817997 K
bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

thd=0x9e6bc18
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
Cannot determine thread, fp=0x52fe5a7c, backtrace may not be correct.
Stack range sanity check OK, backtrace follows:
0x810ac98
0x40043618
0x52fe5e3d
0x8133d8a
0x8132fdd
0x8119330
0x811d3c6
0x8118242
0x8117d42
0x8117668
0x4003e2b6
0x420de407
New value of fp=(nil) failed sanity check, terminating stack trace!
Please read http://dev.mysql.com/doc/mysql/en/Using_stack_trace.html and follow instructions on how to resolve the stack trace. Resolved
stack trace is much more helpful in diagnosing the problem, so please do
resolve it
Trying to get some variables.
Some pointers may be invalid and cause the dump to abort...
thd->query at 0x9e78568 = select pol_no,stk_no,outdate,agtno, retsts,stksts from compulsory.stkmas where pol_no='UPP-G008540'
thd->thread_id=24
The manual page at http://www.mysql.com/doc/en/Crashing.html contains
information that should help you find out what is causing the crash.

How to repeat:
MySQL server is Crash.

This is mysql error log file.
----------------------------------------------------------------------------------------------
/usr/sbin/mysqld-max: ready for connections.
Version: '4.0.26-Max-log' socket: '/var/lib/mysql/mysql.sock' port: 3306 Official MySQL RPM
mysqld got signal 11;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help diagnose
the problem, but since we have already crashed, something is definitely wrong
and this may fail.

key_buffer_size=209715200
read_buffer_size=1044480
max_used_connections=119
max_connections=300
threads_connected=120
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_connections = 817997 K
bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

thd=0x9e6bc18
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
Cannot determine thread, fp=0x52fe5a7c, backtrace may not be correct.
Stack range sanity check OK, backtrace follows:
0x810ac98
0x40043618
0x52fe5e3d
0x8133d8a
0x8132fdd
0x8119330
0x811d3c6
0x8118242
0x8117d42
0x8117668
0x4003e2b6
0x420de407
New value of fp=(nil) failed sanity check, terminating stack trace!
Please read http://dev.mysql.com/doc/mysql/en/Using_stack_trace.html and follow instructions on how to resolve the stack trace. Resolved
stack trace is much more helpful in diagnosing the problem, so please do
resolve it
Trying to get some variables.
Some pointers may be invalid and cause the dump to abort...
thd->query at 0x9e78568 = select pol_no,stk_no,outdate,agtno, retsts,stksts from compulsory.stkmas where pol_no='UPP-G008540'
thd->thread_id=24
The manual page at http://www.mysql.com/doc/en/Crashing.html contains
information that should help you find out what is causing the crash.
----------------------------------------------------------------------------------------------

Then I used command => resolve_stack_dump
This's output. I don't known what it mean. What happen with my server and what should I do to solve this problem.
----------------------------------------------------------------------------------------------
0x810ac98 handle_segfault + 456
0x40043618 _end + 935691984
0x52fe5e3d _end + 1254076149
0x8133d8a _Z12mysql_selectP3THDP13st_table_listR4ListI4ItemEPS4_P8st_orderS9_S7_S9_mP13select_result + 3498
0x8132fdd _Z13handle_selectP3THDP6st_lexP13select_result + 173
0x8119330 _Z21mysql_execute_commandv + 960
0x811d3c6 _Z11mysql_parseP3THDPcj + 150
0x8118242 _Z16dispatch_command19enum_server_commandP3THDPcj + 1202
0x8117d42 _Z10do_commandP3THD + 114
0x8117668 handle_one_connection + 856
0x4003e2b6 _end + 935670638
0x420de407 _end + 969880767
----------------------------------------------------------------------------------------------

Thank for help me.
Wachana S.

Thank you for a problem report. Please send the SHOW CREATE TABLE and SHOW TABLE STATUS results for that compulsory.stkmas table mentioned in the query that lead to crash.

In Today (2006-03-07 08:34) MySQL Server crash again. but not same table.
It is mysql error log file 
---------------------------------------------------------------------------------------------
mysqld got signal 11;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help diagnose
the problem, but since we have already crashed, something is definitely wrong
and this may fail.

key_buffer_size=209715200
read_buffer_size=1044480
max_used_connections=9
max_connections=300
threads_connected=10
It is possible that mysqld could use up to 
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_connections = 817997 K
bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

thd=0x9c67818
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
Cannot determine thread, fp=0x41e5a72c, backtrace may not be correct.
Stack range sanity check OK, backtrace follows:
0x810ac98
0x40043618
0x4207335b
0x4207335b
0x833f083
0x833f6ac
0x81652f0
0x813560d
0x8135d53
0x8133919
0x8132fdd
0x8119330
0x811d3c6
0x8118242
0x8117d42
0x8117668
0x4003e2b6
0x420de407
New value of fp=(nil) failed sanity check, terminating stack trace!
Please read http://dev.mysql.com/doc/mysql/en/Using_stack_trace.html and follow instructions on how to resolve the stack trace. Resolved
stack trace is much more helpful in diagnosing the problem, so please do 
resolve it
Trying to get some variables.
Some pointers may be invalid and cause the dump to abort...
thd->query at 0x9c78178 = select insuno,clmno,brnclmno,acddate,evenresult,acdcase,acdloc,acdamper,acdprovine,clmsts  from ClaimDB.acdmas where insuno = 'M-2061-03/001969' and sts <> '0' order by acddate
thd->thread_id=3
The manual page at http://www.mysql.com/doc/en/Crashing.html contains
information that should help you find out what is causing the crash.
060307  8:26:38 /usr/sbin/mysqld-max: Normal shutdown

A mysqld process already exists at  Tue Mar 7 08:28:05 ICT 2006
060307 08:34:41  mysqld started
060307  8:34:43  InnoDB: Database was not shut down normally.
InnoDB: Starting recovery from log files...
InnoDB: Starting log scan based on checkpoint at
InnoDB: log sequence number 18 559985493
InnoDB: Doing recovery: scanned up to log sequence number 18 559985493
InnoDB: Last MySQL binlog file position 0 27618, file name ./chm2svr-bin.090
060307  8:34:43  InnoDB: Flushing modified pages from the buffer pool...
060307  8:34:43  InnoDB: Started
/usr/sbin/mysqld-max: ready for connections.
Version: '4.0.26-Max-log'  socket: '/var/lib/mysql/mysql.sock'  port: 3306  Official MySQL RPM
---------------------------------------------------------------------------------------------

Then I use resolve_stack_dump command again 
This output from used command.
---------------------------------------------------------------------------------------------
0x810ac98 handle_segfault + 456
0x40043618 _end + 935691984
0x4207335b _end + 969442323
0x4207335b _end + 969442323
0x833f083 my_malloc + 35
0x833f6ac alloc_root + 220
0x81652f0 _ZN10SQL_SELECT17test_quick_selectEP3THDmmmb + 544
0x813560d _Z22get_quick_record_countP3THDP10SQL_SELECTP8st_tablemm + 77
0x8135d53 _Z20make_join_statisticsP4JOINP13st_table_listP4ItemP16st_dynamic_array + 1811
0x8133919 _Z12mysql_selectP3THDP13st_table_listR4ListI4ItemEPS4_P8st_orderS9_S7_S9_mP13select_result + 2361
0x8132fdd _Z13handle_selectP3THDP6st_lexP13select_result + 173
0x8119330 _Z21mysql_execute_commandv + 960
0x811d3c6 _Z11mysql_parseP3THDPcj + 150
0x8118242 _Z16dispatch_command19enum_server_commandP3THDPcj + 1202
0x8117d42 _Z10do_commandP3THD + 114
0x8117668 handle_one_connection + 856
0x4003e2b6 _end + 935670638
0x420de407 _end + 969880767
---------------------------------------------------------------------------------------------

Thank 
Wachana S.

It restart again

0x810ac98 handle_segfault + 456
0x40043618 _end + 935691984
(nil)
0x812df39 _Z11lock_tablesP3THDP13st_table_list + 137
0x812de9f _Z20open_and_lock_tablesP3THDP13st_table_list + 63
0x8119249 _Z21mysql_execute_commandv + 729
0x811d3c6 _Z11mysql_parseP3THDPcj + 150
0x8118242 _Z16dispatch_command19enum_server_commandP3THDPcj + 1202
0x8117d42 _Z10do_commandP3THD + 114
0x8117668 handle_one_connection + 856
0x4003e2b6 _end + 935670638
0x420de407 _end + 969880767

What happen ?????

Please, sent the information I asked you about. For this new crash I need the same information about the ClaimDB.acdmas table.

# Table "acdmas" DDL

CREATE TABLE `acdmas` (
  `clmno` varchar(30) NOT NULL default '',
  `regno` varchar(30) NOT NULL default '',
  `brnclmno` varchar(30) NOT NULL default '',
  `ptype` char(1) NOT NULL default '0',
  `insuno` varchar(30) NOT NULL default '',
  `grpref` varchar(25) NOT NULL default '',
  `itmgrpref` int(4) default '0',
  `clmsts` char(1) default NULL,
  `opensts` char(2) NOT NULL default '',
  `flgcls` char(1) default '0',
  `clsno` char(2) NOT NULL default '',
  `pdtotal` int(4) default NULL,
  `bitotal` int(4) default NULL,
  `acddate` date default '1957-01-01',
  `acdloc` varchar(50) default NULL,
  `acdamper` varchar(25) default NULL,
  `acdprovine` char(2) default NULL,
  `acdcnty` char(1) default '0',
  `evenresult` char(2) default NULL,
  `kfkflg` decimal(1,0) default '1',
  `rcsts` char(1) NOT NULL default '0',
  `rcdetail` varchar(255) NOT NULL default '',
  `rcamt` decimal(11,2) NOT NULL default '0.00',
  `rccls` char(1) NOT NULL default '0',
  `drvflg` decimal(1,0) default '0',
  `drvitem` char(1) default NULL,
  `drvbirth` date default '1857-01-01',
  `drvsex` char(1) default NULL,
  `drvocc` varchar(4) NOT NULL default '',
  `drvidc` varchar(13) default NULL,
  `drvlicn` varchar(8) default NULL,
  `acdtype` char(3) NOT NULL default '',
  `acdcase` varchar(80) default NULL,
  `clmemp` varchar(20) default NULL,
  `clsdate` date default '1957-01-01',
  `regdate` date default '1957-01-01',
  `paydate` date default '1957-01-01',
  `svcno` varchar(5) NOT NULL default '',
  `svcregion` char(2) NOT NULL default '',
  `entuser` varchar(20) NOT NULL default '',
  `entdate` datetime default '1957-01-01 00:00:00',
  `enthost` varchar(20) NOT NULL default '',
  `moduser` varchar(20) NOT NULL default '',
  `moddate` datetime default '1957-01-01 00:00:00',
  `modhost` varchar(20) NOT NULL default '',
  `sts` char(1) default '1',
  `estdate` date default '1957-01-01',
  `odest` decimal(11,2) default '0.00',
  `odamt` decimal(11,2) default '0.00',
  `pdest` decimal(11,2) default '0.00',
  `pdamt` decimal(11,2) default '0.00',
  `biest` decimal(11,2) default '0.00',
  `biamt` decimal(11,2) default '0.00',
  `paest` decimal(11,2) default '0.00',
  `paamt` decimal(11,2) default '0.00',
  `mcest` decimal(11,2) default '0.00',
  `mcamt` decimal(11,2) default '0.00',
  `bbest` decimal(11,2) default '0.00',
  `bbamt` decimal(11,2) default '0.00',
  `allest` decimal(13,2) default '0.00',
  `allamt` decimal(13,2) default '0.00',
  `podest` decimal(11,2) NOT NULL default '0.00',
  `podamt` decimal(11,2) NOT NULL default '0.00',
  `ppdest` decimal(11,2) NOT NULL default '0.00',
  `ppdamt` decimal(11,2) NOT NULL default '0.00',
  `pbiest` decimal(11,2) NOT NULL default '0.00',
  `pbiamt` decimal(11,2) NOT NULL default '0.00',
  `ppaest` decimal(11,2) NOT NULL default '0.00',
  `ppaamt` decimal(11,2) NOT NULL default '0.00',
  `pmcest` decimal(11,2) NOT NULL default '0.00',
  `pmcamt` decimal(11,2) NOT NULL default '0.00',
  `pbbest` decimal(11,2) NOT NULL default '0.00',
  `pbbamt` decimal(11,2) NOT NULL default '0.00',
  `pallest` decimal(13,2) NOT NULL default '0.00',
  `pallamt` decimal(13,2) NOT NULL default '0.00',
  `docid` varchar(20) NOT NULL default '',
  `sex` char(1) default 'M',
  `remark` varchar(200) default NULL,
  `remarkin` varchar(255) NOT NULL default '',
  `remarkit` varchar(255) NOT NULL default '',
  `flgtid` char(1) default '0',
  `odupddate` date default '1957-01-01',
  `odupdfrom` char(1) default '0',
  `pdupddate` date default '1957-01-01',
  `pdupdfrom` char(1) default '0',
  `biupddate` date default '1957-01-01',
  `biupdfrom` char(1) default '0',
  `adupddate` date default '1957-01-01',
  `adupdfrom` char(1) default '0',
  `rcupddate` date default '1957-01-01',
  `rcupdfrom` char(1) default '0',
  `acdupddate` date default '1957-01-01',
  `cplclmno` varchar(30) NOT NULL default '',
  `cplbrnclmno` varchar(30) NOT NULL default '',
  `estfax` decimal(13,2) default '0.00',
  `odfax` decimal(13,2) default '0.00',
  `pdfax` decimal(13,2) default '0.00',
  `bifax` decimal(13,2) default '0.00',
  `pafax` decimal(13,2) default '0.00',
  `mcfax` decimal(13,2) default '0.00',
  `bbfax` decimal(13,2) default '0.00',
  `item` double NOT NULL default '0',
  `detailno` int(5) unsigned NOT NULL default '0',
  `odosd` decimal(13,2) default '0.00',
  `biosd` decimal(13,2) default '0.00',
  `adosd` decimal(13,2) default '0.00',
  `pdosd` decimal(13,2) default '0.00',
  `allosd` decimal(13,2) default '0.00',
  `balest` decimal(13,2) default '0.00',
  `balosd` decimal(13,2) default '0.00',
  `odbalest` decimal(13,2) default '0.00',
  `odbalosd` decimal(13,2) default '0.00',
  `pdbalest` decimal(13,2) default '0.00',
  `pdbalosd` decimal(13,2) default '0.00',
  `bibalest` decimal(13,2) default '0.00',
  `bibalosd` decimal(13,2) default '0.00',
  `adbalest` decimal(13,2) default '0.00',
  `adbalosd` decimal(13,2) default '0.00',
  `modflg` char(1) default '0',
  `paylock` char(1) NOT NULL default '0',
  PRIMARY KEY  (`clmno`),
  KEY `insuno` (`insuno`),
  KEY `grpref` (`grpref`),
  KEY `itmgrpref` (`itmgrpref`),
  KEY `clmsts` (`clmsts`),
  KEY `acddate` (`acddate`),
  KEY `acdamper` (`acdamper`),
  KEY `acdprovine` (`acdprovine`),
  KEY `evenresult` (`evenresult`),
  KEY `drvitem` (`drvitem`),
  KEY `drvocc` (`drvocc`),
  KEY `acdtype` (`acdtype`),
  KEY `svcno` (`svcno`),
  KEY `svcregion` (`svcregion`),
  KEY `docid` (`docid`)
) TYPE=InnoDB

Show Table Status 
Name = ClaimDB.acdmas
Type = InnoDB
Row_Format = Dynamic
Rows = 33333
Avg_row_length = 1592
Data_length = 53067776
Index_length = 23199744

Best Regards
Wachana S.

I had tried to repeat the crash you described with your table and some random data on 4.0.27-BK, but was not able:

mysql> select
    -> insuno,clmno,brnclmno,acddate,evenresult,acdcase,acdloc,acdamper,acdprov
ine,clmsts  from
    -> acdmas where insuno = 'M-2061-03/001969' and sts <> '0' order by acddate;
+------------------+------------------+----------+------------+------------+---------+--------+----------+------------+--------+
| insuno           | clmno            | brnclmno | acddate    | evenresult | acdcase | acdloc | acdamper | acdprovine | clmsts |
+------------------+------------------+----------+------------+------------+---------+--------+----------+------------+--------+
| M-2061-03/001969 | 0.1780669532512  |          | 1957-01-01 | NULL       | NULL    | NULL   | NULL     | NULL       | NULL   |
| M-2061-03/001969 | 0.94944348367811 |          | 1957-01-01 | NULL       | NULL    | NULL   | NULL     | NULL       | NULL   |
+------------------+------------------+----------+------------+------------+---------+--------+----------+------------+--------+
2 rows in set (0.02 sec)

So, do you have any ideas on how to repeat these crashes you described? Can it be a hardware failures related?

No feedback was provided for this bug for over a month, so it is
being suspended automatically. If you are able to provide the
information that was originally requested, please do so and change
the status of the bug back to "Open".