Description:
MySQL restart frequently
2016-06-09 17:27:53 0x7fcc72536700 InnoDB: Assertion failure in thread 140516068124416 in file buf0buf.cc line 3958
InnoDB: Failing assertion: counter < 10
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: http://dev.mysql.com/doc/refman/5.7/en/forcing-innodb-recovery.html
InnoDB: about forcing recovery.
10:27:53 UTC - mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
Attempting to collect some information that could help diagnose the problem.
As this is a crash and something is definitely wrong, the information
collection process might fail.
key_buffer_size=8388608
read_buffer_size=131072
max_used_connections=12
max_threads=600
thread_count=10
connection_count=10
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 1321788 K bytes of memory
Hope that's ok; if not, decrease some variables in the equation.
Thread pointer: 0x7fcbf4000ae0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 7fcc72535e28 thread_stack 0x40000
/usr/sbin/mysqld(my_print_stacktrace+0x35)[0xf4c845]
/usr/sbin/mysqld(handle_fatal_signal+0x4a4)[0x7dd584]
/lib64/libpthread.so.0[0x390c80f790]
/lib64/libc.so.6(gsignal+0x35)[0x390c432625]
/lib64/libc.so.6(abort+0x175)[0x390c433e05]
/usr/sbin/mysqld(_Z18ut_print_timestampP8_IO_FILE+0x0)[0x7cc107]
/usr/sbin/mysqld[0x116cb9e]
/usr/sbin/mysqld(_Z24btr_search_guess_on_hashP12dict_index_tP12btr_search_tPK8dtuple_tmmP9btr_cur_tmP5mtr_t+0x7f7)[0x1164297]
/usr/sbin/mysqld(_Z27btr_cur_search_to_nth_levelP12dict_index_tmPK8dtuple_t15page_cur_mode_tmP9btr_cur_tmPKcmP5mtr_t+0x1081)[0x1159181]
/usr/sbin/mysqld[0x10beac0]
/usr/sbin/mysqld(_Z15row_search_mvccPh15page_cur_mode_tP14row_prebuilt_tmm+0x1c2c)[0x10c0d3c]
/usr/sbin/mysqld(_ZN11ha_innobase10index_readEPhPKhj16ha_rkey_function+0x249)[0xfd5339]
/usr/sbin/mysqld(_ZN7handler17ha_index_read_mapEPhPKhm16ha_rkey_function+0x160)[0x829c40]
/usr/sbin/mysqld[0xcebcfc]
/usr/sbin/mysqld(_Z10sub_selectP4JOINP7QEP_TABb+0x2ba)[0xcf086a]
/usr/sbin/mysqld(_ZN4JOIN4execEv+0x27a)[0xcef7fa]
/usr/sbin/mysqld(_ZN30subselect_single_select_engine4execEv+0x1e0)[0xc22980]
/usr/sbin/mysqld(_ZN14Item_subselect4execEv+0x116)[0xc24966]
/usr/sbin/mysqld(_ZN21Item_exists_subselect8val_boolEv+0x15)[0xc21445]
/usr/sbin/mysqld(_ZN13Item_func_not7val_intEv+0x1f)[0x85509f]
/usr/sbin/mysqld(_ZN4Item8val_boolEv+0xac)[0x8399ec]
/usr/sbin/mysqld(_ZN13Item_cond_and7val_intEv+0x3a)[0x85782a]
/usr/sbin/mysqld[0xceeef4]
/usr/sbin/mysqld(_Z10sub_selectP4JOINP7QEP_TABb+0x323)[0xcf08d3]
/usr/sbin/mysqld[0xcef0ec]
/usr/sbin/mysqld(_Z10sub_selectP4JOINP7QEP_TABb+0x323)[0xcf08d3]
/usr/sbin/mysqld[0xcef0ec]
/usr/sbin/mysqld(_Z10sub_selectP4JOINP7QEP_TABb+0x323)[0xcf08d3]
/usr/sbin/mysqld(_ZN4JOIN4execEv+0x27a)[0xcef7fa]
/usr/sbin/mysqld(_Z12handle_queryP3THDP3LEXP12Query_resultyy+0x250)[0xd59620]
/usr/sbin/mysqld(_ZN21Sql_cmd_insert_select7executeEP3THD+0x3dc)[0xe9fa5c]
/usr/sbin/mysqld(_Z21mysql_execute_commandP3THDb+0xfb9)[0xd1bbb9]
/usr/sbin/mysqld(_ZN13sp_instr_stmt9exec_coreEP3THDPj+0xa1)[0xca05e1]
/usr/sbin/mysqld(_ZN12sp_lex_instr23reset_lex_and_exec_coreEP3THDPjb+0x3bc)[0xca0acc]
/usr/sbin/mysqld(_ZN12sp_lex_instr29validate_lex_and_execute_coreEP3THDPjb+0xb4)[0xca2e74]
/usr/sbin/mysqld(_ZN13sp_instr_stmt7executeEP3THDPj+0x128)[0xca31f8]
/usr/sbin/mysqld(_ZN7sp_head7executeEP3THDb+0x886)[0xc9b7b6]
/usr/sbin/mysqld(_ZN7sp_head17execute_procedureEP3THDP4ListI4ItemE+0x80b)[0xc9c37b]
/usr/sbin/mysqld(_Z21mysql_execute_commandP3THDb+0x1c29)[0xd1c829]
/usr/sbin/mysqld(_Z11mysql_parseP3THDP12Parser_state+0x3ed)[0xd1ffcd]
/usr/sbin/mysqld(_Z16dispatch_commandP3THDPK8COM_DATA19enum_server_command+0x111e)[0xd2115e]
/usr/sbin/mysqld(_Z10do_commandP3THD+0x194)[0xd21dc4]
/usr/sbin/mysqld(handle_connection+0x29c)[0xdf25ec]
/usr/sbin/mysqld(pfs_spawn_thread+0x174)[0x1252c04]
/lib64/libpthread.so.0[0x390c807a51]
/lib64/libpthread.so.0[0x390c807a51]
/lib64/libc.so.6(clone+0x6d)[0x390c4e896d]
Trying to get some variables.
Some pointers may be invalid and cause the dump to abort.
Query (7fcbf4035e10): is an invalid pointer
Connection ID (thread ID): 617
Status: NOT_KILLED
The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you find out what is causing the crash.
2016-06-09T10:27:53.160112Z mysqld_safe Number of processes running now: 0
2016-06-09T10:27:53.162046Z mysqld_safe mysqld restarted
--------------
my.cnf
[mysqld]
#
# Remove leading # and set to the amount of RAM for the most important data
# cache in MySQL. Start at 70% of total RAM for dedicated server, else 10%.
# innodb_buffer_pool_size = 128M
#
# Remove leading # to turn on a very important data integrity option: logging
# changes to the binary log between backups.
# log_bin
#
# Remove leading # to set options mainly useful for reporting servers.
# The server defaults are faster for transactions and fast SELECTs.
# Adjust sizes as needed, experiment to find the optimal values.
#innodb_force_recovery = 1
#innodb_buffer_pool_size = 8192M
innodb_buffer_pool_size = 6144M
query_cache_size = 2048M
join_buffer_size = 128M
sort_buffer_size = 2M
read_rnd_buffer_size = 2M
# Thanh add
innodb_page_cleaners = 8
query_cache_type=ON
#datadir=/var/lib/mysql
datadir=/u02/mysql
socket=/var/lib/mysql/mysql.sock
# Disabling symbolic-links is recommended to prevent assorted security risks
symbolic-links=0
#max_connections = 1000
# Thanh modified Jun 9, 2016
max_connections = 600
user=mysql
large-pages
[mysqld]
innodb_use_native_aio = 1
sql_mode = STRICT_TRANS_TABLES,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION
#connect_timeout=50
# Allow processing table name like Windows (case-insensitive)
lower_case_table_names=1
# Enable Binary Log
server-id=1
log_bin=BSS_log_bin
binlog-do-db=kplusidentitydb
expire_logs_days=5
log_bin_trust_function_creators=1
[mysqld_safe]
log-error=/var/log/mysqld.log
pid-file=/var/run/mysqld/mysqld.pid
#core_file_size = unlimited
#core-file
back_log=65534
How to repeat:
wait for mysql running 1-2 hours
Suggested fix:
I saw free memory decrease...
Please check and advise
