Bug #66508 Slaves continually crash
Submitted: 22 Aug 2012 23:50 Modified: 7 Oct 2012 17:56
Reporter: Michael Finch Email Updates:
Status: No Feedback Impact on me:
None 
Category:MySQL Server: Replication Severity:S1 (Critical)
Version:5.5.25a OS:Any
Assigned to: CPU Architecture:Any
Tags: crash, relay-log, slave

[22 Aug 2012 23:50] Michael Finch
Description:
2 of our 4 slaves got stuck on some statements that replicated from master. This caused the slaves to cycle-crash until we moved master.info. This also caused a lot of relay logs to be written within a very short span of time

-rw-rw---- 1 mysql mysql       191 Aug 22 14:06 relay-log.002058
-rw-rw---- 1 mysql mysql 520242341 Aug 22 16:09 relay-log.002059
-rw-rw---- 1 mysql mysql       150 Aug 22 16:10 relay-log.002060
-rw-rw---- 1 mysql mysql    866270 Aug 22 16:10 relay-log.002061
-rw-rw---- 1 mysql mysql       150 Aug 22 16:10 relay-log.002062
-rw-rw---- 1 mysql mysql       251 Aug 22 16:10 relay-log.002063
-rw-rw---- 1 mysql mysql       150 Aug 22 16:10 relay-log.002064
-rw-rw---- 1 mysql mysql     27954 Aug 22 16:10 relay-log.002065
-rw-rw---- 1 mysql mysql       150 Aug 22 16:10 relay-log.002066
-rw-rw---- 1 mysql mysql    107151 Aug 22 16:10 relay-log.002067
-rw-rw---- 1 mysql mysql       150 Aug 22 16:10 relay-log.002068
-rw-rw---- 1 mysql mysql    126841 Aug 22 16:10 relay-log.002069
-rw-rw---- 1 mysql mysql       150 Aug 22 16:10 relay-log.002070
-rw-rw---- 1 mysql mysql       150 Aug 22 16:10 relay-log.002071
-rw-rw---- 1 mysql mysql   1679220 Aug 22 16:11 relay-log.002072
-rw-rw---- 1 mysql mysql       150 Aug 22 16:11 relay-log.002073
-rw-rw---- 1 mysql mysql   3749381 Aug 22 16:11 relay-log.002074
-rw-rw---- 1 mysql mysql       150 Aug 22 16:11 relay-log.002075
-rw-rw---- 1 mysql mysql   3264994 Aug 22 16:11 relay-log.002076
-rw-rw---- 1 mysql mysql       150 Aug 22 16:11 relay-log.002077

23:09:27 UTC - mysqld got signal 11 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.

key_buffer_size=67108864
read_buffer_size=1048576
max_used_connections=55
max_threads=300
thread_count=3
connection_count=3
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 683355 K  bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0x1f37a550
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 497caa78 thread_stack 0x40000
/usr/libexec/mysqld(my_print_stacktrace+0x2e)[0x7af76e]
/usr/libexec/mysqld(handle_fatal_signal+0x3e2)[0x67c372]
/lib64/libpthread.so.0[0x2ae123b72be0]
/usr/libexec/mysqld(_ZN14Arg_comparator27compare_int_unsigned_signedEv+0x12)[0x69f7a2]
/usr/libexec/mysqld(_ZN12Item_func_eq7val_intEv+0x2e)[0x6a94ce]
/usr/libexec/mysqld(_ZN4Item8val_boolEv+0x5c)[0x68b13c]
/usr/libexec/mysqld(_ZN13Item_cond_and7val_intEv+0x2e)[0x6a327e]
/usr/libexec/mysqld(_Z12mysql_deleteP3THDP10TABLE_LISTP4ItemP10SQL_I_ListI8st_orderEyy+0x898)[0x771278]
/usr/libexec/mysqld(_Z21mysql_execute_commandP3THD+0x3d5c)[0x57771c]
/usr/libexec/mysqld(_Z11mysql_parseP3THDPcjP12Parser_state+0x12a)[0x579a9a]
/usr/libexec/mysqld(_ZN15Query_log_event14do_apply_eventEPK14Relay_log_infoPKcj+0xc57)[0x7348f7]
/usr/libexec/mysqld(_Z26apply_event_and_update_posP9Log_eventP3THDP14Relay_log_info+0x16e)[0x5170be]
/usr/libexec/mysqld[0x51e721]
/usr/libexec/mysqld(handle_slave_sql+0xc46)[0x51fa36]
/lib64/libpthread.so.0[0x2ae123b6a77d]
/lib64/libc.so.6(clone+0x6d)[0x2ae1255dd25d]

Trying to get some variables.
Some pointers may be invalid and cause the dump to abort.
Query (2aab611c5b96): is an invalid pointer
Connection ID (thread ID): 2
Status: NOT_KILLED

The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you find out what is causing the crash.
120822 16:09:35 mysqld_safe Number of processes running now: 0
120822 16:09:35 mysqld_safe mysqld restarted
120822 16:09:35 [Note] Plugin 'FEDERATED' is disabled.
120822 16:09:35 InnoDB: The InnoDB memory heap is disabled
120822 16:09:35 InnoDB: Mutexes and rw_locks use GCC atomic builtins
120822 16:09:35 InnoDB: Compressed tables use zlib 1.2.3
120822 16:09:35 InnoDB: Using Linux native AIO
120822 16:09:35 InnoDB: Initializing buffer pool, size = 256.0M
120822 16:09:35 InnoDB: Completed initialization of buffer pool
120822 16:09:35 InnoDB: highest supported file format is Barracuda.
InnoDB: Log scan progressed past the checkpoint lsn 1558687660793
120822 16:09:35  InnoDB: Database was not shut down normally!
InnoDB: Starting crash recovery.
InnoDB: Reading tablespace information from the .ibd files...
InnoDB: Restoring possible half-written data pages from the doublewrite
InnoDB: buffer...
InnoDB: Doing recovery: scanned up to log sequence number 1558689892412
120822 16:10:40  InnoDB: Starting an apply batch of log records to the database...
InnoDB: Progress in percents: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 $
InnoDB: Apply batch completed
InnoDB: Last MySQL binlog file position 0 1065500205, file name /mnt/mysql/log/bin-log.000921
120822 16:10:41  InnoDB: Waiting for the background threads to start
120822 16:10:42 InnoDB: 1.1.8 started; log sequence number 1558689892412
120822 16:10:42 [Note] Recovering after a crash using /mnt/mysql/log/bin-log
120822 16:10:48 [Note] Starting crash recovery...
120822 16:10:48 [Note] Crash recovery finished.
120822 16:10:48 [Note] Server hostname (bind-address): '0.0.0.0'; port: 3306
120822 16:10:48 [Note]   - '0.0.0.0' resolves to '0.0.0.0';
120822 16:10:48 [Note] Server socket created on IP: '0.0.0.0'.

How to repeat:
Slaves were replicating normally from master. Nothing unusual happened, and other slaves did not experience this issue.
[23 Aug 2012 5:55] Valeriy Kravchuk
Can you try to get crashing statement from the binary log?
[27 Aug 2012 17:37] Michael Finch
The slaves crashed here. I removed the master.info on the slaves, ran this update manually, and then pointed them to the next position in the binlog and they were fine.

# at 559560948
#120822 16:19:56 server id 1  end_log_pos 559561022     Query   thread_id=39552912      exec_time=0     error_code=0
SET TIMESTAMP=1345677596/*!*/;
BEGIN
/*!*/;                        
# at 559561022
#120822 16:19:56 server id 1  end_log_pos 559561509     Query   thread_id=39552912      exec_time=0     error_code=0
SET TIMESTAMP=1345677596/*!*/;         
UPDATE weekly_keywordgroup_overview_universal SET
num_keywords_universal_across_domain='{"70": 0, "40": 0, "10": 0, "75": 0, "79": 0, "50": 5, "20": 0, "-1": 5, "60": 0, "30": 0}',
num_keywords_universal='{"70": 0, "40": 0, "10": 0, "75": 0, "79": 0, "50": 5, "20": 0, "-1": 5, "60": 0, "30": 0}',
num_keywords=7
WHERE account_id=4727 AND search_engine_id=34 AND domain_id=113 AND week=201226 AND keyword_group_id=7524
/*!*/;
# at 559561509
#120822 16:19:56 server id 1  end_log_pos 559561536     Xid = 280411802
COMMIT/*!*/;

Thanks
[6 Sep 2012 18:52] Sveta Smirnova
Thank you for the feedback.

Please provide output of SHOW CREATE TABLE weekly_keywordgroup_overview_universal
[6 Sep 2012 20:51] Michael Finch
mysql> SHOW CREATE TABLE weekly_keywordgroup_overview_universal\G
*************************** 1. row ***************************
       Table: weekly_keywordgroup_overview_universal
Create Table: CREATE TABLE `weekly_keywordgroup_overview_universal` (
  `id` int(11) unsigned NOT NULL AUTO_INCREMENT,
  `account_id` int(6) unsigned NOT NULL,
  `week` int(6) unsigned NOT NULL,
  `search_engine_id` int(4) unsigned DEFAULT NULL,
  `domain_id` int(11) unsigned NOT NULL,
  `keyword_group_id` int(11) NOT NULL,
  `num_keywords_universal_across_domain` varchar(200) DEFAULT NULL,
  `num_keywords_universal` varchar(200) DEFAULT NULL,
  `num_keywords` int(11) DEFAULT '0',
  PRIMARY KEY (`id`,`week`),
  UNIQUE KEY `unique_index` (`week`,`account_id`,`search_engine_id`,`domain_id`,`keyword_group_id`) USING HASH
) ENGINE=InnoDB AUTO_INCREMENT=6294348 DEFAULT CHARSET=latin1
/*!50100 PARTITION BY RANGE (week)
(PARTITION p0 VALUES LESS THAN (201122) ENGINE = InnoDB,
 PARTITION p1 VALUES LESS THAN (201123) ENGINE = InnoDB,
 PARTITION p2 VALUES LESS THAN (201124) ENGINE = InnoDB,
 PARTITION p3 VALUES LESS THAN (201125) ENGINE = InnoDB,
 PARTITION p4 VALUES LESS THAN (201126) ENGINE = InnoDB,
 PARTITION p5 VALUES LESS THAN (201127) ENGINE = InnoDB,
 PARTITION p6 VALUES LESS THAN (201128) ENGINE = InnoDB,
 PARTITION p7 VALUES LESS THAN (201129) ENGINE = InnoDB,
 PARTITION p8 VALUES LESS THAN (201130) ENGINE = InnoDB,
 PARTITION p9 VALUES LESS THAN (201131) ENGINE = InnoDB,
 PARTITION p10 VALUES LESS THAN (201132) ENGINE = InnoDB,
 PARTITION p11 VALUES LESS THAN (201133) ENGINE = InnoDB,
 PARTITION p12 VALUES LESS THAN (201134) ENGINE = InnoDB,
 PARTITION p13 VALUES LESS THAN (201135) ENGINE = InnoDB,
 PARTITION p14 VALUES LESS THAN (201136) ENGINE = InnoDB,
 PARTITION p15 VALUES LESS THAN (201137) ENGINE = InnoDB,
 PARTITION p16 VALUES LESS THAN (201138) ENGINE = InnoDB,
 PARTITION p17 VALUES LESS THAN (201139) ENGINE = InnoDB,
 PARTITION p18 VALUES LESS THAN (201140) ENGINE = InnoDB,
 PARTITION p19 VALUES LESS THAN (201141) ENGINE = InnoDB,
 PARTITION p20 VALUES LESS THAN (201142) ENGINE = InnoDB,
 PARTITION p21 VALUES LESS THAN (201143) ENGINE = InnoDB,
 PARTITION p22 VALUES LESS THAN (201144) ENGINE = InnoDB,
 PARTITION p23 VALUES LESS THAN (201145) ENGINE = InnoDB,
 PARTITION p24 VALUES LESS THAN (201146) ENGINE = InnoDB,
 PARTITION p25 VALUES LESS THAN (201147) ENGINE = InnoDB,
 PARTITION p26 VALUES LESS THAN (201148) ENGINE = InnoDB,
 PARTITION p27 VALUES LESS THAN (201149) ENGINE = InnoDB,
 PARTITION p28 VALUES LESS THAN (201150) ENGINE = InnoDB,
 PARTITION p29 VALUES LESS THAN (201151) ENGINE = InnoDB,
 PARTITION p30 VALUES LESS THAN (201152) ENGINE = InnoDB,
 PARTITION p31 VALUES LESS THAN (201201) ENGINE = InnoDB,
 PARTITION p32 VALUES LESS THAN (201202) ENGINE = InnoDB,
 PARTITION p33 VALUES LESS THAN (201203) ENGINE = InnoDB,
 PARTITION p34 VALUES LESS THAN (201204) ENGINE = InnoDB,
 PARTITION p35 VALUES LESS THAN (201205) ENGINE = InnoDB,
 PARTITION p36 VALUES LESS THAN (201206) ENGINE = InnoDB,
 PARTITION p37 VALUES LESS THAN (201207) ENGINE = InnoDB,
 PARTITION p38 VALUES LESS THAN (201208) ENGINE = InnoDB,
 PARTITION p39 VALUES LESS THAN (201209) ENGINE = InnoDB,
 PARTITION p40 VALUES LESS THAN (201210) ENGINE = InnoDB,
 PARTITION p41 VALUES LESS THAN (201211) ENGINE = InnoDB,
 PARTITION p42 VALUES LESS THAN (201212) ENGINE = InnoDB,
 PARTITION p43 VALUES LESS THAN (201213) ENGINE = InnoDB,
 PARTITION p44 VALUES LESS THAN (201214) ENGINE = InnoDB,
 PARTITION p45 VALUES LESS THAN (201215) ENGINE = InnoDB,
 PARTITION p46 VALUES LESS THAN (201216) ENGINE = InnoDB,
 PARTITION p47 VALUES LESS THAN (201217) ENGINE = InnoDB,
 PARTITION p48 VALUES LESS THAN (201218) ENGINE = InnoDB,
 PARTITION p49 VALUES LESS THAN (201219) ENGINE = InnoDB,
 PARTITION p50 VALUES LESS THAN (201220) ENGINE = InnoDB,
 PARTITION p51 VALUES LESS THAN (201221) ENGINE = InnoDB,
 PARTITION p52 VALUES LESS THAN (201222) ENGINE = InnoDB,
 PARTITION p53 VALUES LESS THAN (201223) ENGINE = InnoDB,
 PARTITION p54 VALUES LESS THAN (201224) ENGINE = InnoDB,
 PARTITION p55 VALUES LESS THAN (201225) ENGINE = InnoDB,
 PARTITION p56 VALUES LESS THAN (201226) ENGINE = InnoDB,
 PARTITION p57 VALUES LESS THAN (201227) ENGINE = InnoDB,
 PARTITION p58 VALUES LESS THAN (201228) ENGINE = InnoDB,
 PARTITION p59 VALUES LESS THAN (201229) ENGINE = InnoDB,
 PARTITION p60 VALUES LESS THAN (201230) ENGINE = InnoDB,
 PARTITION p61 VALUES LESS THAN (201231) ENGINE = InnoDB,
 PARTITION p62 VALUES LESS THAN (201232) ENGINE = InnoDB,
 PARTITION p63 VALUES LESS THAN (201233) ENGINE = InnoDB,
 PARTITION p64 VALUES LESS THAN (201234) ENGINE = InnoDB,
 PARTITION p65 VALUES LESS THAN (201235) ENGINE = InnoDB,
 PARTITION p66 VALUES LESS THAN (201236) ENGINE = InnoDB,
 PARTITION p67 VALUES LESS THAN (201237) ENGINE = InnoDB,
 PARTITION p68 VALUES LESS THAN (201238) ENGINE = InnoDB,
 PARTITION p69 VALUES LESS THAN (201239) ENGINE = InnoDB,
 PARTITION p70 VALUES LESS THAN (201240) ENGINE = InnoDB,
 PARTITION p71 VALUES LESS THAN (201241) ENGINE = InnoDB,
 PARTITION p72 VALUES LESS THAN (201242) ENGINE = InnoDB,
 PARTITION p73 VALUES LESS THAN (201243) ENGINE = InnoDB,
 PARTITION p74 VALUES LESS THAN (201244) ENGINE = InnoDB,
 PARTITION p75 VALUES LESS THAN (201245) ENGINE = InnoDB,
 PARTITION p76 VALUES LESS THAN (201246) ENGINE = InnoDB,
 PARTITION p77 VALUES LESS THAN (201247) ENGINE = InnoDB,
 PARTITION p78 VALUES LESS THAN (201248) ENGINE = InnoDB,
 PARTITION p79 VALUES LESS THAN (201249) ENGINE = InnoDB,
 PARTITION p80 VALUES LESS THAN (201250) ENGINE = InnoDB,
 PARTITION p81 VALUES LESS THAN (201251) ENGINE = InnoDB,
 PARTITION p82 VALUES LESS THAN (201252) ENGINE = InnoDB,
 PARTITION p83 VALUES LESS THAN (201301) ENGINE = InnoDB,
 PARTITION p84 VALUES LESS THAN (201302) ENGINE = InnoDB,
 PARTITION p85 VALUES LESS THAN (201303) ENGINE = InnoDB,
 PARTITION p86 VALUES LESS THAN (201304) ENGINE = InnoDB,
 PARTITION p87 VALUES LESS THAN (201305) ENGINE = InnoDB,
 PARTITION p88 VALUES LESS THAN (201306) ENGINE = InnoDB,
 PARTITION p89 VALUES LESS THAN (201307) ENGINE = InnoDB,
 PARTITION pmax VALUES LESS THAN MAXVALUE ENGINE = InnoDB) */
1 row in set (0.00 sec)
[7 Sep 2012 17:56] Sveta Smirnova
Thank you for the feedback.

I can not repeat described behavior. Please try with current version 5.5.27 and inform us if problem still exists.
[8 Oct 2012 1:00] Bugs System
No feedback was provided for this bug for over a month, so it is
being suspended automatically. If you are able to provide the
information that was originally requested, please do so and change
the status of the bug back to "Open".