MySQL Bugs: #72287: mysqld crash during REORGANIZE PARTITION

Bug #72287	mysqld crash during REORGANIZE PARTITION
Submitted:	9 Apr 2014 9:02	Modified:	16 May 2018 15:17
Reporter:	Conor Murphy	Email Updates:
Status:	No Feedback	Impact on me:	None
Category:	MySQL Server: Partitions	Severity:	S1 (Critical)
Version:	5.6.16-log	OS:	Solaris (s10x_u10wos_17b)
Assigned to:		CPU Architecture:	Any

Description:
During a maintance window, we alter the partitioning of a number of tables. We are now getting mysqld crashing during the execution of the ALTER commands

2014-04-08 22:46:35 1291 [ERROR] /opt/mysql/mysql/bin/mysqld: Incorrect key file for table './statsdb/hires_disk_stat#P#P20130501#TMP#.MYI'; try to repair it
2014-04-08 22:46:35 1291 [ERROR] Got an error from thread_id=2720, /export/home2/pb2/build/sb_0-11248666-1389788591.3/mysql-5.6.16/storage/myisam/mi_write.c:226
2014-04-08 22:46:35 1291 [ERROR] MySQL thread id 2720, OS thread handle 0x24, query id 50109 localhost statsadm System lock
ALTER TABLE hires_disk_stat REORGANIZE PARTITION P20130401,P20130501 INTO ( PARTITION P20130501 VALUES LESS THAN (TO_DAYS('2013-05-01')) )
21:46:47 UTC - mysqld got signal 11 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed, 
something is definitely wrong and this may fail.

key_buffer_size=402653184
read_buffer_size=2097152
max_used_connections=49
max_threads=151
thread_count=7
connection_count=7
It is possible that mysqld could use up to 
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 1013739 K  bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0x3e4e1350
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
/opt/mysql/mysql/bin/mysqld:my_print_stacktrace+0x1e
/opt/mysql/mysql/bin/mysqld:handle_fatal_signal+0x305
/lib/amd64/libc.so.1:0xdd6b6
/lib/amd64/libc.so.1:0xd1f82
/opt/mysql/mysql/bin/mysqld:mi_extra+0x1d [ Signal 11 (SEGV)]
/opt/mysql/mysql/bin/mysqld:__1cJha_myisamFextra6MnRha_extra_function__i_+0x28
/opt/mysql/mysql/bin/mysqld:__1cMha_partitionQloop_extra_alter6MnRha_extra_function__i_+0x36
/opt/mysql/mysql/bin/mysqld:__1cMha_partitionFextra6MnRha_extra_function__i_+0x13b
/opt/mysql/mysql/bin/mysqld:__1cYwait_while_table_is_used6FpnDTHD_pnFTABLE_nRha_extra_function__b_+0x77
/opt/mysql/mysql/bin/mysqld:__1cXhandle_alter_part_error6FpnSst_lock_param_type_bbbb_v_+0x6e
/opt/mysql/mysql/bin/mysqld:__1cbAfast_alter_partition_table6FpnDTHD_pnFTABLE_pnKAlter_info_pnYst_ha_create_information_pnKTABLE_LIST_pcpkc_I_+0x567
/opt/mysql/mysql/bin/mysqld:__1cRmysql_alter_table6FpnDTHD_pc2pnYst_ha_create_information_pnKTABLE_LIST_pnKAlter_info_IpnIst_order_b_b_+0x1741
/opt/mysql/mysql/bin/mysqld:__1cTSql_cmd_alter_tableHexecute6MpnDTHD__b_+0x49a
/opt/mysql/mysql/bin/mysqld:__1cVmysql_execute_command6FpnDTHD__i_+0x5452
/opt/mysql/mysql/bin/mysqld:__1cLmysql_parse6FpnDTHD_pcIpnMParser_state__v_+0x319
/opt/mysql/mysql/bin/mysqld:__1cQdispatch_command6FnTenum_server_command_pnDTHD_pcI_b_+0xd20
/opt/mysql/mysql/bin/mysqld:__1cKdo_command6FpnDTHD__b_+0xca
/opt/mysql/mysql/bin/mysqld:__1cYdo_handle_one_connection6FpnDTHD__v_+0x157
/opt/mysql/mysql/bin/mysqld:handle_one_connection+0x47
/opt/mysql/mysql/bin/mysqld:pfs_spawn_thread+0x14c
/lib/amd64/libc.so.1:0xdd37b
/lib/amd64/libc.so.1:0xdd5b0
Please read http://dev.mysql.com/doc/refman/5.1/en/resolve-stack-dump.html
and follow instructions on how to resolve the stack trace.
Resolved stack trace is much more helpful in diagnosing the
problem, so please do resolve it

Trying to get some variables.
Some pointers may be invalid and cause the dump to abort.
Query (3d73a9d8): ALTER TABLE hires_disk_stat REORGANIZE PARTITION P20130401,P20130501 INTO ( PARTITION P20130501 VALUES LESS THAN (TO_DAYS('2013-05-01')) )
Connection ID (thread ID): 2720
Status: NOT_KILLED

$ uname -srpi
SunOS 5.10 i386 i86pc

How to repeat:
Use ALTER TABLE REORGANIZE PARTITION

What is the table structure?

SHOW CREATE TABLE hires_disk_stat \G

mysql> SHOW CREATE TABLE hires_disk_stat\G
*************************** 1. row ***************************
       Table: hires_disk_stat
Create Table: CREATE TABLE `hires_disk_stat` (
  `time` datetime NOT NULL,
  `diskid` mediumint(8) unsigned NOT NULL,
  `busy` tinyint(3) unsigned NOT NULL,
  `rws` int(10) unsigned NOT NULL,
  `blks` int(10) unsigned NOT NULL,
  `avque` decimal(4,1) unsigned NOT NULL,
  `avwait` decimal(5,1) unsigned NOT NULL,
  `avserv` decimal(5,1) unsigned NOT NULL,
  KEY `myidx` (`time`,`diskid`)
) ENGINE=MyISAM DEFAULT CHARSET=latin1
/*!50100 PARTITION BY RANGE ( TO_DAYS(time))
(PARTITION P20070101 VALUES LESS THAN (733042) ENGINE = MyISAM,
 PARTITION P20080101 VALUES LESS THAN (733407) ENGINE = MyISAM,
 PARTITION P20090101 VALUES LESS THAN (733773) ENGINE = MyISAM,
 PARTITION P20100101 VALUES LESS THAN (734138) ENGINE = MyISAM,
 PARTITION P20110101 VALUES LESS THAN (734503) ENGINE = MyISAM,
 PARTITION P20120101 VALUES LESS THAN (734868) ENGINE = MyISAM,
 PARTITION P20130101 VALUES LESS THAN (735234) ENGINE = MyISAM,
 PARTITION P20130401 VALUES LESS THAN (735324) ENGINE = MyISAM,
 PARTITION P20130501 VALUES LESS THAN (735354) ENGINE = MyISAM,
 PARTITION P20130601 VALUES LESS THAN (735385) ENGINE = MyISAM,
 PARTITION P20130701 VALUES LESS THAN (735415) ENGINE = MyISAM,
 PARTITION P20130801 VALUES LESS THAN (735446) ENGINE = MyISAM,
 PARTITION P20130901 VALUES LESS THAN (735477) ENGINE = MyISAM,
 PARTITION P20131001 VALUES LESS THAN (735507) ENGINE = MyISAM,
 PARTITION P20131101 VALUES LESS THAN (735538) ENGINE = MyISAM,
 PARTITION P20131201 VALUES LESS THAN (735568) ENGINE = MyISAM,
 PARTITION P20140101 VALUES LESS THAN (735599) ENGINE = MyISAM,
 PARTITION P20140201 VALUES LESS THAN (735630) ENGINE = MyISAM,
 PARTITION P20140301 VALUES LESS THAN (735658) ENGINE = MyISAM,
 PARTITION P20140401 VALUES LESS THAN (735689) ENGINE = MyISAM,
 PARTITION P20140402 VALUES LESS THAN (735690) ENGINE = MyISAM,
 PARTITION P20140403 VALUES LESS THAN (735691) ENGINE = MyISAM,
 PARTITION P20140404 VALUES LESS THAN (735692) ENGINE = MyISAM,
 PARTITION P20140405 VALUES LESS THAN (735693) ENGINE = MyISAM,
 PARTITION P20140406 VALUES LESS THAN (735694) ENGINE = MyISAM,
 PARTITION P20140407 VALUES LESS THAN (735695) ENGINE = MyISAM,
 PARTITION P20140408 VALUES LESS THAN (735696) ENGINE = MyISAM,
 PARTITION P20140409 VALUES LESS THAN (735697) ENGINE = MyISAM,
 PARTITION PMAXVALUE VALUES LESS THAN MAXVALUE ENGINE = MyISAM) */
1 row in set (0.03 sec)

mysql>

/opt/mysql/mysql/bin/mysqld: Incorrect key file for table './statsdb/hires_disk_stat#P#P20130501#TMP#.MYI'; try to repair it

Also make sure enough disk space is available...

Should be enough disk space, I haven't seen any messages about the file system being full

$ df -h /data/db 
Filesystem             size   used  avail capacity  Mounted on
datapool/data/db       3.9T   1.4T   405G    79%    /data/db
$ 

The partitions being merged have ~ 10.5GB (MYD+MYI)
$ ls -lth hires_disk_stat* | egrep 'P20130401|P20130501'
-rw-rw----   1 mysql    mysql       1.0G Mar 11 15:14 hires_disk_stat#P#P20130501.MYI
-rw-rw----   1 mysql    mysql       3.2G Mar 11 15:14 hires_disk_stat#P#P20130401.MYI
-rw-rw----   1 mysql    mysql       4.5G Mar  8 06:48 hires_disk_stat#P#P20130401.MYD
-rw-rw----   1 mysql    mysql       1.8G Feb 11 03:42 hires_disk_stat#P#P20130501.MYD

It was done the repair table and tested with new releases. Thanks.

No feedback was provided for this bug for over a month, so it is
being suspended automatically. If you are able to provide the
information that was originally requested, please do so and change
the status of the bug back to "Open".