Bug #46338 Unknow to crash MySQL
Submitted: 22 Jul 2009 10:49 Modified: 14 Sep 2009 19:56
Reporter: Steven Yeung Email Updates:
Status: Not a Bug Impact on me:
None 
Category:MySQL Server Severity:S2 (Serious)
Version:5.1.34-community OS:Linux (RHEL 5)
Assigned to: CPU Architecture:Any

[22 Jul 2009 10:49] Steven Yeung
Description:
I got below's message in the error log today. But I don't know the problem.

 mysqld got signal 11 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help diagnose
the problem, but since we have already crashed, something is definitely wrong
and this may fail.

key_buffer_size=402653184
read_buffer_size=2097152
max_used_connections=61
max_threads=2048
threads_connected=61
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 406240 K
bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

thd: 0x96615938
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0x969ed210 thread_stack 0x60000
/usr/sbin/mysqld(my_print_stacktrace+0x21)[0x848bb11]
/usr/sbin/mysqld(handle_segfault+0x37f)[0x81e30af]
[0x4b4420]
/lib/libc.so.6(__stpcpy+0x13)[0x17d763]
/usr/sbin/mysqld[0x8281993]
/usr/sbin/mysqld(_Z23check_routine_level_aclP3THDPKcS2_b+0x6b)[0x8281b4b]
/usr/sbin/mysqld(_Z25check_some_routine_accessP3THDPKcS2_b+0x93)[0x81ecaa3]
/usr/sbin/mysqld(_Z17store_schema_procP3THDP8st_tableS2_PKcbS4_+0x74a)[0x82eb87a]
/usr/sbin/mysqld(_Z16fill_schema_procP3THDP10TABLE_LISTP4Item+0x200)[0x82ebbb0]
/usr/sbin/mysqld(_Z24get_schema_tables_resultP4JOIN23enum_schema_table_state+0x1bb)[0x82e475b]
/usr/sbin/mysqld(_ZN4JOIN4execEv+0x479)[0x825bc89]
/usr/sbin/mysqld(_Z12mysql_selectP3THDPPP4ItemP10TABLE_LISTjR4ListIS1_ES2_jP8st_orderSB_S2_SB_yP13select_resultP18st_select_lex_unitP13st_select_lex+0x137)[0x825df07]
/usr/sbin/mysqld(_Z13handle_selectP3THDP6st_lexP13select_resultm+0x146)[0x825e7c6]
/usr/sbin/mysqld[0x81ee43f]
/usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x4d89)[0x81f5369]
/usr/sbin/mysqld(_Z11mysql_parseP3THDPKcjPS2_+0x1ee)[0x81f7f3e]
/usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcj+0x114a)[0x81f90aa]
/usr/sbin/mysqld(_Z10do_commandP3THD+0xe1)[0x81f9631]
/usr/sbin/mysqld(handle_one_connection+0x7b1)[0x81ea6b1]
/lib/libpthread.so.0[0x5712db]
/lib/libc.so.6(clone+0x5e)[0x1dd14e]
Trying to get some variables.
Some pointers may be invalid and cause the dump to abort...
thd->query at 0xa9c9ef0 = SHOW PROCEDURE STATUS
thd->thread_id=315
thd->killed=NOT_KILLED

How to repeat:
I don't know
[22 Jul 2009 21:16] Sveta Smirnova
Thank you for the report.

We have 2 similar, but not exactly same bugs: bug #34895 and bug #41726.

Did you do upgrade? Did you run mysql_upgrade? If you have general and/or binary query log turned on please send them to us, so we can guess which queries were running same time. Please also try current version 5.1.36 to check if problem fixed occasionally.
[23 Jul 2009 7:21] Steven Yeung
Yes. I have exceuted mysql_upgrade command before.
And I have added below's setting in my.cnf two weeks ago.

max_sp_recursion_depth = 255
thread_stack = 393216
[27 Jul 2009 20:47] Sveta Smirnova
Thank you for the feedback.

I can not repeat described behavior using load only with statements from the binary log provided (update, insert, delete). Please try to gather more information about load from your application: probably some statements affect this. You can temporarily turn on general query log (as it is online since version 5.1) or use logging from your application or MySQL Proxy. Also, please, provide your configuration file and output of SHOW GRANTS for user you run SHOW PROCEDURE STATUS as.
[28 Jul 2009 1:29] Steven Yeung
Actually, I try to comment the new configration. Then, the crash doesn't occur any more. But I need stored procedure to support recursion here. So, I hope belows information can help yours to find out the problem.

My Server information:

CPU: Intel(R) Xeon(R) CPU           E5335  @ 2.00GHz
RAM: 2G RAM
OS: RHEL 5

My configration as below:

[mysqld]
#default#datadir=/var/lib/mysql
#default#socket=/var/lib/mysql/mysql.sock
# Default to using old password format for compatibility with mysql 3.x
# clients (those using the mysqlclient10 compatibility package).
#default#old_passwords=1
server-id=1
max_connections=2048
default-storage-engine=MYISAM
##log
log-bin=lx18-bin
#log-slow-queries
#--log-bin=lx18-bin
#--log-bin=/var/lib/mysql/lx18-bin (not worked)
character-set-server=big5
key_buffer = 384M
read_buffer_size = 2M
sort_buffer_size = 2M
myisam_sort_buffer_size = 64M
table_cache = 512
thread_cache_size = 8
query_cache_size = 32M
# Try number of CPU's*2 for thread_concurrency
thread_concurrency = 8
max_sp_recursion_depth = 255
#thread stack in 384KB
thread_stack = 393216

[mysql.server]
#default#user=mysql
#default#basedir=/var/lib

[client]
##default-character-set=big5

[mysqld_safe]
#default#err-log=/var/mysql/log/mysqld.log
#default#pid-file=/var/run/mysqld/mysqld.pid
##server-id=1
##max_connections=2048
##log
##log-bin
##log-slow-queries
[28 Jul 2009 5:52] Sveta Smirnova
Thank you for the feedback.

Please also provide output of SHOW GRANTS for user you run SHOW PROCEDURE STATUS as.
[29 Jul 2009 1:06] Steven Yeung
There are two users:

GRANT ALL PRIVILEGES ON *.* TO 'admin'@'%' IDENTIFIED BY PASSWORD <<PASSWORD>> WITH GRANT OPTION

GRANT SELECT, INSERT, UPDATE, DELETE, CREATE, DROP, RELOAD, SHUTDOWN, FILE, REFERENCES, INDEX, ALTER, SHOW DATABASES, CREATE TEMPORARY TABLES, LOCK TABLES, EXECUTE, REPLICATION SLAVE, REPLICATION CLIENT, CREATE VIEW, SHOW VIEW, CREATE ROUTINE, ALTER ROUTINE ON *.* TO 'keith'@'%' IDENTIFIED BY PASSWORD <<PASSWORD>>
[4 Sep 2009 7:46] Sveta Smirnova
Thank you for the feedback.

I still can not repeat described behavior. Do you see crash every time when issue SHOW PROCEDURE STATUS? Please try current version 5.1.38 and if problem still exists provide dump of mysql.proc table.
[14 Sep 2009 15:14] Steven Yeung
I guess the problem was caused by old stored procedure (from old MySQL version).
All old stored procedure show in mysql.proc column 'db' is empty. After I delete those stored procedure, it doesn't occurs any more.
[14 Sep 2009 19:56] Sveta Smirnova
Thank you for the feedback.

Closed as "Not a Bug", because last comment: empty record in mysql.proc can lead to unpredictable behavior.