Description:
Highloaded mysql server started to crash randomly during the day.
error log:
-------------------------------
pure virtual method called
terminate called without an active exception
120411 18:44:06 - mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help diagnose
the problem, but since we have already crashed, something is definitely wrong
and this may fail.
key_buffer_size=536870912
read_buffer_size=4194304
max_used_connections=69
max_threads=1000
thread_count=32
connection_count=32
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 135704303 K
bytes of memory
Hope that's ok; if not, decrease some variables in the equation.
Thread pointer: 0x2aab1a9d9470
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0x40c920d0 thread_stack 0x40000
/usr/local/mysql55/libexec/mysqld(my_print_stacktrace+0x33)[0x762e73]
/usr/local/mysql55/libexec/mysqld(handle_segfault+0x36e)[0x4ed49e]
/lib64/libpthread.so.0[0x2aebd0067b10]
/lib64/libc.so.6(gsignal+0x35)[0x2aebd1290265]
/lib64/libc.so.6(abort+0x110)[0x2aebd1291d10]
/usr/lib64/libstdc++.so.6(_ZN9__gnu_cxx27__verbose_terminate_handlerEv+0x114)[0x2aebd0b8cd14]
/usr/lib64/libstdc++.so.6[0x2aebd0b8ae16]
/usr/lib64/libstdc++.so.6[0x2aebd0b8ae43]
/usr/lib64/libstdc++.so.6[0x2aebd0b8b34f]
/usr/local/mysql55/libexec/mysqld[0x66e121]
/usr/local/mysql55/libexec/mysqld(_ZN14Item_func_case18fix_length_and_decEv+0x8b)[0x679e8b]
/usr/local/mysql55/libexec/mysqld(_ZN9Item_func10fix_fieldsEP3THDPP4Item+0x17b)[0x69102b]
/usr/local/mysql55/libexec/mysqld(_ZN14Item_func_case10fix_fieldsEP3THDPP4Item+0x19)[0x671119]
/usr/local/mysql55/libexec/mysqld(_Z12setup_fieldsP3THDPP4ItemR4ListIS1_E17enum_mark_columnsPS5_b+0x14c)[0x52112c]
/usr/local/mysql55/libexec/mysqld(_Z12mysql_updateP3THDP10TABLE_LISTR4ListI4ItemES6_PS4_jP8st_ordery15enum_duplicatesbPySB_+0x39e)[0x5be51e]
/usr/local/mysql55/libexec/mysqld(_Z21mysql_execute_commandP3THD+0x2cf9)[0x55a509]
/usr/local/mysql55/libexec/mysqld(_ZN13sp_instr_stmt9exec_coreEP3THDPj+0x1c)[0x712afc]
/usr/local/mysql55/libexec/mysqld(_ZN13sp_lex_keeper23reset_lex_and_exec_coreEP3THDPjbP8sp_instr+0x94)[0x712c64]
/usr/local/mysql55/libexec/mysqld(_ZN13sp_instr_stmt7executeEP3THDPj+0x641)[0x718561]
/usr/local/mysql55/libexec/mysqld(_ZN7sp_head7executeEP3THDb+0x4c9)[0x716d29]
/usr/local/mysql55/libexec/mysqld(_ZN7sp_head17execute_procedureEP3THDP4ListI4ItemE+0x4b5)[0x717535]
/usr/local/mysql55/libexec/mysqld(_Z21mysql_execute_commandP3THD+0x893)[0x5580a3]
/usr/local/mysql55/libexec/mysqld(_Z11mysql_parseP3THDPcjP12Parser_state+0xfe)[0x55d56e]
/usr/local/mysql55/libexec/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcj+0x1353)[0x55e923]
/usr/local/mysql55/libexec/mysqld(_Z10do_commandP3THD+0xc2)[0x55ec52]
/usr/local/mysql55/libexec/mysqld(_Z24do_handle_one_connectionP3THD+0x2f4)[0x5e9784]
/usr/local/mysql55/libexec/mysqld(handle_one_connection+0x4a)[0x5e986a]
/lib64/libpthread.so.0[0x2aebd005f73d]
/lib64/libc.so.6(clone+0x6d)[0x2aebd13344bd]
Trying to get some variables.
Some pointers may be invalid and cause the dump to abort.
Query (0x2aab12944c68): is an invalid pointer
Connection ID (thread ID): 6397
Status: NOT_KILLED
The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you find out what is causing the crash.
------------------------------------
resolved stack trace:
# /usr/local/mysql55/bin/resolve_stack_dump -s /root/mysqld.nm -n stack.txt
0x762e73 my_safe_print_str + 387
0x4ed49e handle_segfault + 1006
0x2aebd0067b10 _end + -820854200
0x2aebd1290265 _end + -801814115
0x2aebd1291d10 _end + -801807288
0x2aebd0b8cd14 _end + -809167796
0x2aebd0b8ae16 _end + -809175730
0x2aebd0b8ae43 _end + -809175685
0x2aebd0b8b34f _end + -809174393
0x66e121 List_iterator<Cached_item>::List_iterator() + 273
0x679e8b Item_func_case::fix_length_and_dec() + 267
0x69102b Item_func_sp::fix_fields(THD*, Item**) + 59
0x671119 Item_cond::Item_cond(THD*, Item_cond*) + 57
0x52112c setup_fields(THD*, Item**, List<Item>&, enum_mark_columns, List<Item>*, bool) + 460
0x5be51e mysql_update(THD*, TABLE_LIST*, List<Item>&, List<Item>&, Item*, unsigned int, st_order*, unsigned long long, enum_duplicates, + 1054
0x55a509 mysql_execute_command(THD*) + 11641
0x712afc sp_instr::exec_open_and_lock_tables(THD*, TABLE_LIST*) + 28
0x712c64 sp_lex_keeper::reset_lex_and_exec_core(THD*, unsigned int*, bool, sp_instr*) + 276
0x718561 sp_instr_stmt::execute(THD*, unsigned int*) + 1729
0x716d29 sp_head::execute(THD*, bool) + 1353
0x717535 sp_head::execute_procedure(THD*, List<Item>*) + 1333
0x5580a3 mysql_execute_command(THD*) + 2323
0x55d56e dispatch_command(enum_server_command, THD*, char*, unsigned int) + 30
0x55e923 dispatch_command(enum_server_command, THD*, char*, unsigned int) + 5075
0x55ec52 execute_init_command(THD*, st_mysql_lex_string*, st_mysql_rwlock*) + 18
0x5e9784 do_handle_one_connection(THD*) + 884
0x5e986a check_for_max_user_connections(THD*, user_conn*) + 106
0x2aebd005f73d _end + -820887947
0x2aebd13344bd _end + -801141771
-----------------
mysql> show variables like '%connections%';
+----------------------+-------+
| Variable_name | Value |
+----------------------+-------+
| max_connections | 1000 |
| max_user_connections | 30 |
+----------------------+-------+
2 rows in set (0.00 sec)
but where are not so many connections, usually 40-50 at the same time.
Where are no any user limits on privileges level.
# mysql -e "show processlist"|wc -l
45
How to repeat:
Don't know. Server have a lot of different user's bases with a lot of queries.
