2016-02-29 13:31:28 28457 [ERROR] Error reading packet from server: Lost connection to MySQL server during query ( server_errno=2013) 2016-02-29 13:31:28 28457 [Note] Slave I/O thread killed while reading event 2016-02-29 13:31:28 28457 [Note] Slave I/O thread exiting, read up to log 'mysql-server-bin.000001', position 35504473 2016-02-29 13:31:28 28457 [Warning] Storing MySQL user name or password information in the master info repository is not secure and is therefore not recommended. Please consider using the USER and PASSWORD connection options for START SLAVE; see the 'START SLAVE Syntax' in the MySQL Manual for more information. 2016-02-29 13:31:28 28457 [Warning] Slave SQL: If a crash happens this configuration does not guarantee that the relay log info will be consistent, Error_code: 0 2016-02-29 13:31:28 28457 [Note] Slave SQL thread initialized, starting replication in log 'mysql-server-bin.000001' at position 28756090, relay log '/[path to]/mysql-server-relay-bin.000004' position: 429 2016-02-29 13:31:28 2ad890df6940 InnoDB: Assertion failure in thread 47109631863104 in file row0mysql.cc line 1700 InnoDB: Failing assertion: node->pcur->rel_pos == BTR_PCUR_ON InnoDB: We intentionally generate a memory trap. InnoDB: Submit a detailed bug report to http://bugs.mysql.com. InnoDB: If you get repeated assertion failures or crashes, even InnoDB: immediately after the mysqld startup, there may be InnoDB: corruption in the InnoDB tablespace. Please refer to InnoDB: http://dev.mysql.com/doc/refman/5.6/en/forcing-innodb-recovery.html InnoDB: about forcing recovery. 13:31:28 UTC - mysqld got signal 6 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. key_buffer_size=16777216 read_buffer_size=131072 max_used_connections=252 max_threads=500 thread_count=12 connection_count=10 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 215227 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0xaf9de190 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 2ad890df5930 thread_stack 0x40000 /usr/sbin/mysqld(my_print_stacktrace+0x35)[0x940ec5] /usr/sbin/mysqld(handle_fatal_signal+0x3e8)[0x6a1fd8] /lib64/libpthread.so.0[0x36d240eca0] /lib64/libc.so.6(gsignal+0x35)[0x36d1c30265] /lib64/libc.so.6(abort+0x110)[0x36d1c31d10] /usr/sbin/mysqld[0xa593d7] /usr/sbin/mysqld[0x9d0b09] /usr/sbin/mysqld(_ZN7handler13ha_update_rowEPKhPh+0xb5)[0x5c0dc5] /usr/sbin/mysqld(_ZN21Update_rows_log_event11do_exec_rowEPK14Relay_log_info+0x11d)[0x8d266d] /usr/sbin/mysqld(_ZN14Rows_log_event12do_apply_rowEPK14Relay_log_info+0x2f)[0x8c9b7f] /usr/sbin/mysqld(_ZN14Rows_log_event24do_table_scan_and_updateEPK14Relay_log_info+0x14d)[0x8d33dd] /usr/sbin/mysqld(_ZN14Rows_log_event14do_apply_eventEPK14Relay_log_info+0xd7c)[0x8d5cdc] /usr/sbin/mysqld(_ZN9Log_event11apply_eventEP14Relay_log_info+0x74)[0x8cf6b4] /usr/sbin/mysqld(_Z26apply_event_and_update_posPP9Log_eventP3THDP14Relay_log_info+0x18a)[0x90761a] /usr/sbin/mysqld[0x917b52] /usr/sbin/mysqld(handle_slave_sql+0xd26)[0x919146] /usr/sbin/mysqld(pfs_spawn_thread+0x126)[0x9ba476] /lib64/libpthread.so.0[0x36d240683d] /lib64/libc.so.6(clone+0x6d)[0x36d1cd526d] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0): is an invalid pointer Connection ID (thread ID): 241123611 Status: NOT_KILLED The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. 160229 13:31:32 mysqld_safe Number of processes running now: 0 160229 13:31:32 mysqld_safe mysqld restarted 2016-02-29 13:31:32 5831 [Warning] Using unique option prefix myisam_recover instead of myisam-recover-options is deprecated and will be removed in a future release. Please use the full name instead. 2016-02-29 13:31:32 5831 [Note] Plugin 'FEDERATED' is disabled. 2016-02-29 13:31:32 5831 [Note] InnoDB: The InnoDB memory heap is disabled 2016-02-29 13:31:32 5831 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2016-02-29 13:31:32 5831 [Note] InnoDB: Compressed tables use zlib 1.2.3 2016-02-29 13:31:32 5831 [Note] InnoDB: Using Linux native AIO 2016-02-29 13:31:32 5831 [Note] InnoDB: Using CPU crc32 instructions 2016-02-29 13:31:32 5831 [Note] InnoDB: Initializing buffer pool, size = 102.0G 2016-02-29 13:31:38 5831 [Note] InnoDB: Completed initialization of buffer pool 2016-02-29 13:31:38 5831 [Note] InnoDB: Highest supported file format is Barracuda. 2016-02-29 13:31:39 5831 [Note] InnoDB: Log scan progressed past the checkpoint lsn 1237725469564 2016-02-29 13:31:39 5831 [Note] InnoDB: Database was not shutdown normally! 2016-02-29 13:31:39 5831 [Note] InnoDB: Starting crash recovery. 2016-02-29 13:31:39 5831 [Note] InnoDB: Reading tablespace information from the .ibd files... 2016-02-29 13:31:39 5831 [Note] InnoDB: Restoring possible half-written data pages 2016-02-29 13:31:39 5831 [Note] InnoDB: from the doublewrite buffer... InnoDB: Doing recovery: scanned up to log sequence number 1237727299383 InnoDB: Transaction 2266721257 was in the XA prepared state. InnoDB: 1 transaction(s) which must be rolled back or cleaned up InnoDB: in total 0 row operations to undo InnoDB: Trx id counter is 2266721536 2016-02-29 13:31:40 5831 [Note] InnoDB: Starting an apply batch of log records to the database... InnoDB: Progress in percent: 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 InnoDB: Apply batch completed InnoDB: Last MySQL binlog file position 0 6143424, file name mysql-server-bin.001212 InnoDB: Starting in background the rollback of uncommitted transactions 2016-02-29 13:31:47 5831 [Note] InnoDB: 128 rollback segment(s) are active. 2016-02-29 13:31:47 2b31386a5940 InnoDB: Rollback of non-prepared transactions completed 2016-02-29 13:31:47 5831 [Note] InnoDB: Waiting for purge to start 2016-02-29 13:31:47 5831 [Note] InnoDB: 5.6.15 started; log sequence number 1237727299383 2016-02-29 13:31:47 5831 [Note] Recovering after a crash using /[path to]/mysql-server-bin 2016-02-29 13:31:47 5831 [Note] Starting crash recovery... 2016-02-29 13:31:47 2b160d1c1590 InnoDB: Starting recovery for XA transactions... 2016-02-29 13:31:47 2b160d1c1590 InnoDB: Transaction 2266721257 in prepared state after recovery 2016-02-29 13:31:47 2b160d1c1590 InnoDB: Transaction contains changes to 1 rows 2016-02-29 13:31:47 2b160d1c1590 InnoDB: 1 transactions in prepared state after recovery 2016-02-29 13:31:47 5831 [Note] Found 1 prepared transaction(s) in InnoDB 2016-02-29 13:31:47 5831 [Note] Crash recovery finished. 2016-02-29 13:31:47 5831 [Warning] Failed to setup SSL 2016-02-29 13:31:47 5831 [Warning] SSL error: SSL_CTX_set_default_verify_paths failed 2016-02-29 13:31:47 5831 [Note] RSA private key file not found: /[path to]//private_key.pem. Some authentication plugins will not work. 2016-02-29 13:31:47 5831 [Note] RSA public key file not found: /[path to]//public_key.pem. Some authentication plugins will not work. 2016-02-29 13:31:47 5831 [Note] Server hostname (bind-address): '0.0.0.0'; port: 3306 2016-02-29 13:31:47 5831 [Note] - '0.0.0.0' resolves to '0.0.0.0'; 2016-02-29 13:31:47 5831 [Note] Server socket created on IP: '0.0.0.0'. 2016-02-29 13:31:47 5831 [ERROR] /usr/sbin/mysqld: Table './mysql/user' is marked as crashed and should be repaired 2016-02-29 13:31:47 5831 [Warning] Checking table: './mysql/user' 2016-02-29 13:31:47 5831 [ERROR] 1 client is using or hasn't closed the table properly 2016-02-29 13:31:47 5831 [ERROR] /usr/sbin/mysqld: Table './mysql/db' is marked as crashed and should be repaired 2016-02-29 13:31:47 5831 [Warning] Checking table: './mysql/db' 2016-02-29 13:31:47 5831 [ERROR] 1 client is using or hasn't closed the table properly 2016-02-29 13:31:47 5831 [Warning] 'user' entry '[replication user]@[host]' ignored in --skip-name-resolve mode. 2016-02-29 13:31:47 5831 [Warning] 'user' entry '[replication user]@[host]' ignored in --skip-name-resolve mode. 2016-02-29 13:31:47 5831 [ERROR] /usr/sbin/mysqld: Table './mysql/tables_priv' is marked as crashed and should be repaired 2016-02-29 13:31:47 5831 [Warning] Checking table: './mysql/tables_priv' 2016-02-29 13:31:47 5831 [ERROR] 1 client is using or hasn't closed the table properly 2016-02-29 13:31:47 5831 [Warning] Recovery from master pos 28756090 and file mysql-server-bin.000001. 2016-02-29 13:31:47 5831 [Note] Event Scheduler: Loaded 0 events 2016-02-29 13:31:47 5831 [Note] /usr/sbin/mysqld: ready for connections.