| Bug #35574 | MySQLd crashing on SHOW STATUS | ||
|---|---|---|---|
| Submitted: | 26 Mar 2008 14:52 | Modified: | 2 Apr 2008 10:11 |
| Reporter: | Dominik Jansen | Email Updates: | |
| Status: | Can't repeat | Impact on me: | |
| Category: | MySQL Server | Severity: | S2 (Serious) |
| Version: | 5.0.54-log | OS: | Linux (2.6.22-gentoo-r8) |
| Assigned to: | Sergei Glukhov | CPU Architecture: | Any |
| Tags: | backtrace, crash, show status | ||
[26 Mar 2008 14:52]
Dominik Jansen
[26 Mar 2008 14:55]
Dominik Jansen
Backtrace file for Bug 35574
Attachment: mysqld.backtrace.bug-35574.txt (text/plain), 840 bytes.
[26 Mar 2008 15:02]
MySQL Verification Team
Thank you for the bug report. I can't repeat the crash reported on latest source server and the server version reported is quite older. Could you please upgrade to latest released version. Thanks in advance.
[27 Mar 2008 8:41]
Dominik Jansen
Hello again, i did the update to MySQL 5.0.54-log because this is marked as stable on my gentoo system. When I run SHOW STATUS the server still crashes. Thank you in advance.
[27 Mar 2008 8:41]
Dominik Jansen
Backtrace file for Bug 35574 with MySQL 5.0.54-log
Attachment: mysqld.backtrace.bug-35574.mysql_5.0.54-log.txt (text/plain), 1.51 KiB.
[27 Mar 2008 14:05]
Heikki Tuuri
This crash happens inside InnoDB:
0x819fbed handle_segfault + 797
0x834508f srv_export_innodb_status + 159
0x824e984 ha_update_statistics() + 20
0x8277437 fill_status(THD*, st_table_list*, Item*) + 55
0x82761d2 get_schema_tables_result(JOIN*, enum_schema_table_state) + 258
0x8200ef3 JOIN::exec() + 2819
0x8202a5d _Z12mysql_selectP3THDPPP4ItemP13st_table_listjR4ListIS1_ES2_jP8st_orderSB_S2_SB_yP13select_resultP18st_select_lex_unitP13st_sel + 285
0x82032e1 handle_select(THD*, st_lex*, select_result*, unsigned long) + 321
0x81b8235 mysql_execute_command(THD*) + 13829
0x81bb450 mysql_parse(THD*, char const*, unsigned int, char const**) + 368
0x81bb92c dispatch_command(enum_server_command, THD*, char*, unsigned int) + 1052
0x81bcc4b do_command(THD*) + 171
0x81bd684 handle_one_connection + 2372
0xb7ddc18b _end + -1351252557
0xb7c0704e _end + -1353173898
Maybe it crashes in mutex_enter(&srv_innodb_monitor_mutex). But that is strange. Very basic code.
/**********************************************************************
Function to pass InnoDB status variables to MySQL */
void
srv_export_innodb_status(void)
{
mutex_enter(&srv_innodb_monitor_mutex);
export_vars.innodb_data_pending_reads= os_n_pending_reads;
export_vars.innodb_data_pending_writes= os_n_pending_writes;
export_vars.innodb_data_pending_fsyncs=
fil_n_pending_log_flushes + fil_n_pending_tablespace_flushes;
export_vars.innodb_data_fsyncs= os_n_fsyncs;
export_vars.innodb_data_read= srv_data_read;
export_vars.innodb_data_reads= os_n_file_reads;
export_vars.innodb_data_writes= os_n_file_writes;
export_vars.innodb_data_written= srv_data_written;
export_vars.innodb_buffer_pool_read_requests= buf_pool->n_page_gets;
export_vars.innodb_buffer_pool_write_requests= srv_buf_pool_write_reque\
sts;
export_vars.innodb_buffer_pool_wait_free= srv_buf_pool_wait_free;
export_vars.innodb_buffer_pool_pages_flushed= srv_buf_pool_flushed;
export_vars.innodb_buffer_pool_reads= srv_buf_pool_reads;
export_vars.innodb_buffer_pool_read_ahead_rnd= srv_read_ahead_rnd;
export_vars.innodb_buffer_pool_read_ahead_seq= srv_read_ahead_seq;
export_vars.innodb_buffer_pool_pages_data= UT_LIST_GET_LEN(buf_pool->LR\
U);
export_vars.innodb_buffer_pool_pages_dirty= UT_LIST_GET_LEN(buf_pool->f\
lush_list);
export_vars.innodb_buffer_pool_pages_free= UT_LIST_GET_LEN(buf_pool->fr\
ee);
export_vars.innodb_buffer_pool_pages_latched= buf_get_latched_pages_num\
ber();
export_vars.innodb_buffer_pool_pages_total= buf_pool->curr_size;
export_vars.innodb_buffer_pool_pages_misc= buf_pool->max_size -
UT_LIST_GET_LEN(buf_pool->LRU) - UT_LIST_GET_LEN(buf_pool->free);
export_vars.innodb_page_size= UNIV_PAGE_SIZE;
export_vars.innodb_log_waits= srv_log_waits;
export_vars.innodb_os_log_written= srv_os_log_written;
export_vars.innodb_os_log_fsyncs= fil_n_log_flushes;
export_vars.innodb_os_log_pending_fsyncs= fil_n_pending_log_flushes;
export_vars.innodb_os_log_pending_writes= srv_os_log_pending_writes;
export_vars.innodb_log_write_requests= srv_log_write_requests;
export_vars.innodb_log_writes= srv_log_writes;
export_vars.innodb_dblwr_pages_written= srv_dblwr_pages_written;
export_vars.innodb_dblwr_writes= srv_dblwr_writes;
export_vars.innodb_pages_created= buf_pool->n_pages_created;
export_vars.innodb_pages_read= buf_pool->n_pages_read;
...
[27 Mar 2008 14:13]
Heikki Tuuri
Dominik, the resolved stack trace below does not make sense. Please resolve it with the right .sym file. 0x819fbed String::set_ascii(char const*, unsigned int) + 125 0x834508f __bam_dpages + 223 0x824e984 simple_pred(Item_func*, Item**, bool*) + 420 0x8277437 READ_INFO::find_start_of_fields() + 87 0x82761d2 mysql_recreate_table(THD*, TABLE_LIST*) + 3506 0x8200ef3 JOIN::optimize() + 6435 0x8202a5d JOIN::optimize() + 13453 0x82032e1 JOIN::optimize() + 15633 0x81b8235 mysql_new_select(st_lex*, bool) + 229 0x81bb450 mysql_execute_command(THD*) + 12080 0x81bb92c mysql_execute_command(THD*) + 13324 0x81bcc4b mysql_execute_command(THD*) + 18219 0x81bd684 mysql_execute_command(THD*) + 20836 Regards, Heikki
[27 Mar 2008 15:01]
Dominik Jansen
Hi, i did this commands: ---------- # /etc/init.d/mysql start * Starting mysql ... * Starting mysql (/etc/mysql/my.cnf) # mysql -p Enter password: Welcome to the MySQL monitor. Commands end with ; or \g. Your MySQL connection id is 7 Server version: 5.0.54-log Gentoo Linux mysql-5.0.54 Type 'help;' or '\h' for help. Type '\c' to clear the buffer. mysql> SHOW STATUS; ERROR 2013 (HY000): Lost connection to MySQL server during query mysql> exit Bye ------------ Then copied this lines out of /var/log/mysql/mysqld.err : ------------ thd=0x87516d8 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... Cannot determine thread, fp=0xb52bcca8, backtrace may not be correct. Stack range sanity check OK, backtrace follows: 0x81a30cd 0x834bc9f 0x8254574 0x827d567 0x827c242 0x8205e01 0x820795d 0x82081e1 0x81baef8 0x81bee1b 0x81bf32f 0x81c1093 0xb7e1218b 0xb7c3d04e New value of fp=(nil) failed sanity check, terminating stack trace! Please read http://dev.mysql.com/doc/mysql/en/using-stack-trace.html and follow instructions on how to resolve the stack trace. Resolved stack trace is much more helpful in diagnosing the problem, so please do resolve it Trying to get some variables. Some pointers may be invalid and cause the dump to abort... thd->query at 0x875a748 = SHOW STATUS thd->thread_id=7 The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. ------------- And pasted it to /tmp/mysqld.stack Then did: ------------- # nm -D -n /usr/sbin/mysqld > /tmp/mysqld.sym # resolve_stack_dump -s /tmp/mysqld.sym -n /tmp/mysqld.stack|c++filt ------------- And resolved: ------------- thd=0x87516d8 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... Cannot determine thread, fp=0xb52bcca8, backtrace may not be correct. Stack range sanity check OK, backtrace follows: 0x81a30cd handle_segfault + 797 0x834bc9f srv_export_innodb_status + 159 0x8254574 ha_update_statistics() + 20 0x827d567 fill_status(THD*, TABLE_LIST*, Item*) + 55 0x827c242 get_schema_tables_result(JOIN*, enum_schema_table_state) + 258 0x8205e01 JOIN::exec() + 2817 0x820795d _Z12mysql_selectP3THDPPP4ItemP10TABLE_LISTjR4ListIS1_ES2_jP8st_orderSB_S2_SB_yP13select_resultP18st_select_lex_unitP13st_select + 285 0x82081e1 handle_select(THD*, st_lex*, select_result*, unsigned long) + 321 0x81baef8 mysql_execute_command(THD*) + 10712 0x81bee1b mysql_parse(THD*, char const*, unsigned int, char const**) + 379 0x81bf32f dispatch_command(enum_server_command, THD*, char*, unsigned int) + 1103 0x81c1093 handle_one_connection + 2563 0xb7e1218b _end + -1351084973 0xb7c3d04e _end + -1353006314 New value of fp=(nil) failed sanity check, terminating stack trace! Please read http://dev.mysql.com/doc/mysql/en/using-stack-trace.html and follow instructions on how to resolve the stack trace. Resolved stack trace is much more helpful in diagnosing the problem, so please do resolve it Trying to get some variables. Some pointers may be invalid and cause the dump to abort... thd->query at 0x875a748 = SHOW STATUS thd->thread_id=7 The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. ------------------ This is what I've done, following the manual on: http://dev.mysql.com/doc/refman/5.0/en/using-stack-trace.html Isn't it right?
[27 Mar 2008 16:30]
Heikki Tuuri
Dominik, yes that backtrace is right. It was the other one that was not sensible. Are you using InnoDB tables? Please post your my.cnf. --Heikki
[28 Mar 2008 6:48]
Dominik Jansen
Hello Heikki, yes I'm using InnoDB tables. I'll attach the my.cnf file after this post. This box is just running the MySQL Server and zabbix with 512MB mem. Thank you in advance, Dominik
[31 Mar 2008 9:41]
Dominik Jansen
Hello again, i finally got the problem. It was a missconfigured innodb_data_file_path: innodb_data_file_path = ibdata1:128M;ibdata250M:autoextend:max:12800M What caused the server to disable innodb? Now i can run 'show status' without crashing ther server ...
[31 Mar 2008 10:04]
Dominik Jansen
Hmm, it looks like InnoDB is still Disabled when I run show engines. When running show innodb status I get: mysql> show innodb status; ERROR 1235 (42000): Cannot call SHOW INNODB STATUS because skip-innodb is defined skip-innodb isn't defined
[31 Mar 2008 12:41]
Heikki Tuuri
Hi! Can MySQL engineers fix this by blocking calls to InnoDB or other disabled engines, even if MySQL sees a .frm file for an InnoDB table? Another option is to return some error code from InnoDB, but I am not sure if MySQL handles that gracefully either. The crash apparently is due to the fact that InnoDB was not initialized at all, and the mutex protecting the InnoDB statistics had not been created. Then mutex_enter() -> crash. Best regards, Heikki
[2 Apr 2008 10:11]
Sergei Glukhov
Checked on latest 5.0 tree, cant repeat. Valeriy, if you are able to repeat this crash, please provide clear 'how to repeat' example.
