Bug #43357 | Server crash when selecting from INFORMATION_SCHEMA | ||
---|---|---|---|
Submitted: | 4 Mar 2009 9:35 | Modified: | 25 Feb 2010 0:19 |
Reporter: | Rafal Somla | Email Updates: | |
Status: | Closed | Impact on me: | |
Category: | MySQL Server: Backup | Severity: | S3 (Non-critical) |
Version: | 6.0,5.4 | OS: | Any |
Assigned to: | Sergei Glukhov | CPU Architecture: | Any |
Tags: | crash, pushbuild, sporadic, test failure |
[4 Mar 2009 9:35]
Rafal Somla
[6 Mar 2009 7:58]
Rafal Somla
Here is another crash I got in the same code area: #3 <signal handler called> #4 0x401d6947 in raise () from /lib/tls/libc.so.6 #5 0x401d80c9 in abort () from /lib/tls/libc.so.6 #6 0x0893d014 in safe_mutex_lock (mp=0x9f293b8, my_flags=0, file=0x8afde58 "sql_show.cc", line=1887) at thr_mutex.c:179 #7 0x084a4c31 in fill_schema_processlist (thd=0x9e53a30, tables=0x9fe2660, cond=0xa1acea8) at sql_show.cc:1887 #8 0x08497c8a in get_schema_tables_result (join=0xa1a7a08, executed_place=PROCESSED_BY_JOIN_EXEC) at sql_show.cc:6480 #9 0x083c47ae in JOIN::exec (this=0xa1a7a08) at sql_select.cc:2393 #10 0x083c0d6d in mysql_select (thd=0x9e53a30, rref_pointer_array=0x9e54e70, tables=0x9fe2660, wild_num=0, fields=@0x9e54e00, conds=0x9fe3010, og_num=0, order=0x0, group=0x0, having=0x0, proc_param=0x0, select_options=2684635648, result=0x9fe3ce0, unit=0x9e54ad4, select_lex=0x9e54d6c) at sql_select.cc:3062 #11 0x083c65d2 in handle_select (thd=0x9e53a30, lex=0x9e54a78, result=0x9fe3ce0, setup_tables_done_option=0) at sql_select.cc:314 #12 0x0832a407 in execute_sqlcom_select (thd=0x9e53a30, all_tables=0x9fe2660) at sql_parse.cc:4817 #13 0x0832b4de in mysql_execute_command (thd=0x9e53a30) at sql_parse.cc:2142 #14 0x08333ab7 in mysql_parse (thd=0x9e53a30, inBuf=0x9fe2098 "SELECT state, info FROM INFORMATION_SCHEMA.PROCESSLIST\nWHERE info LIKE \"BACKUP DATABASE%\"\n OR info LIKE \"RESTORE FROM%\"\n OR info LIKE \"CREATE DATABASE bml_test_db1%\"\n OR info LIKE \"DROP TA"..., length=208, found_semicolon=0x5ec30f20) at sql_parse.cc:5812 Crash is due to abort() call here: 173 if (!mp->file) 174 { 175 fprintf(stderr, 176 "safe_mutex: Trying to lock unitialized mutex at %s, line %d\n", 177 file, line); 178 fflush(stderr); 179 abort(); <---------------- 180 } (gdb) p *mp $2 = {global = {__m_reserved = 0, __m_count = 0, __m_owner = 0x0, __m_kind = 2, __m_lock = {__status = 0, __spinlock = 0}}, mutex = {__m_reserved = 0, __m_count = 0, __m_owner = 0x0, __m_kind = 3, __m_lock = {__status = 0, __spinlock = 0}}, file = 0x0, name = 0x8a77825 "LOCK_delete", line = 799, count = 0, create_flags = 2, active_flags = 0, id = 53824, thread = 0, locked_mutex = 0xa186748, used_mutex = 0xa186780, prev = 0x0, next = 0x0} This is in a call to safe_mutex_lock() with file="sql_show.cc" and line=1887. I obtained this crash after running "mtr backup_bml" several times in an infinite loop.
[24 Mar 2009 9:04]
Bugs System
A patch for this bug has been committed. After review, it may be pushed to the relevant source trees for release in the next version. You can access the patch from: http://lists.mysql.com/commits/70145 2792 Rafal Somla 2009-03-23 Remove SELECTs from I_S.PROCESSLIST from tests, because such selects are unstable in the current server (see bug#37990, bug#41346, bug#43357). These selects increase feedback given by test and thus increases chances of detecting problems, but are not essential for test purposes. @ mysql-test/suite/backup/include/bml_test.inc Comment-out SELECTs from I_S.PROCESSLIST. @ mysql-test/suite/backup/t/backup_bml.test Comment-out SELECTs from I_S.PROCESSLIST. @ mysql-test/suite/backup_engines/include/backup_restore_interrupt.inc Comment-out SELECTs from I_S.PROCESSLIST.
[24 Mar 2009 9:10]
Rafal Somla
Moving back to verified - the patch does not fix this bug, only modifies test cases to not use I_S selects.
[24 Mar 2009 10:39]
Bugs System
A patch for this bug has been committed. After review, it may be pushed to the relevant source trees for release in the next version. You can access the patch from: http://lists.mysql.com/commits/70165 2793 Rafal Somla 2009-03-23 Remove SELECTs from I_S.PROCESSLIST from tests, because such selects are unstable in the current server (see bug#37990, bug#41346, bug#43357). These selects increase feedback given by test and thus increases chances of detecting problems, but are not essential for test purposes. @ mysql-test/suite/backup/include/bml_test.inc Comment-out SELECTs from I_S.PROCESSLIST. @ mysql-test/suite/backup/t/backup_bml.test Comment-out SELECTs from I_S.PROCESSLIST. @ mysql-test/suite/backup_engines/include/backup_restore_interrupt.inc Comment-out SELECTs from I_S.PROCESSLIST.
[24 Mar 2009 10:48]
Rafal Somla
Not fixed yet.
[23 Apr 2009 7:19]
Bugs System
Pushed into 6.0.11-alpha (revid:alik@sun.com-20090423070920-e5lq3vrrqi016z2c) (version source revid:alik@sun.com-20090423070920-e5lq3vrrqi016z2c) (merge vers: 6.0.11-alpha) (pib:6)
[30 Apr 2009 12:45]
Rafal Somla
I tried it on the recent clone of our main-6.0 tree (6.0.11-alpha, revid:jperkin@sun.com-20090423215644-h7ssug9w1hdgzn39). After enabling selects from INFORMATION_SCHEMA in suite/backup/include/bml_test.inc I get crash each time I run backup_bml test.
[17 Jun 2009 8:47]
Bugs System
A patch for this bug has been committed. After review, it may be pushed to the relevant source trees for release in the next version. You can access the patch from: http://lists.mysql.com/commits/76427 3367 Sergey Glukhov 2009-06-17 Bug#43357 Server crash when selecting from INFORMATION_SCHEMA The crash happens due to thd->mysys_var which is cleaned up in 'delete thd'. 'delete thd' should be called under LOCK_thread_count lock but it does not happen for backup thread. The fix is: -move 'delete thd' under LOCK_thread_count lock. -removed unnecessary net_end() call as it is performed in 'delete thd' Note: backup_bml.result is not updated as it reqires FALCON engine which does not work at the moment. @ mysql-test/suite/backup/include/bml_test.inc uncomment I_S.processlist test @ mysql-test/suite/backup/r/backup_intr_errors.result result fix @ mysql-test/suite/backup/t/backup_bml.test uncomment I_S.processlist test @ mysql-test/suite/backup_engines/include/backup_restore_interrupt.inc uncomment I_S.processlist test @ sql/backup/be_thread.cc -move 'delete thd' under LOCK_thread_count lock. -removed unnecessary net_end() call as it is performed in 'delete thd'.
[27 Jul 2009 15:57]
Ingo Strüwing
Some time ago I sent some questions by email. I set this back to "in progress" to get it from my todo list until the questions are answered.
[25 Nov 2009 10:22]
Philip Stoev
Grammar file for bug 43357
Attachment: bug43357.yy (application/octet-stream, text), 179 bytes.
[25 Nov 2009 10:24]
Philip Stoev
To reproduce within a few seconds: perl runall.pl \ --grammar=conf/bug43357.yy \ --mysqld=--mysql-backup \ --basedir=/build/bzr/6.0-codebase-bugfixing This particular test uses non-concurrent BACKUP/RESTORE and concurrent SHOW PROCESSLIST, but this crash has been observed when SHOW PROCESSLIST was ran only once every 10 seconds. This tells me that there is a big opportunity for crashes in production systems if a monitoring tool is running and backups are being taken.
[2 Dec 2009 15:56]
Bugs System
A patch for this bug has been committed. After review, it may be pushed to the relevant source trees for release in the next version. You can access the patch from: http://lists.mysql.com/commits/92511 3733 Sergey Glukhov 2009-12-02 Bug#43357 Server crash when selecting from INFORMATION_SCHEMA Server crashes in mysqld_list_processes() function on pthread_mutex_unlock(&tmp->LOCK_thd_data) call. It happens because threads after its delete are still visible in thread list with destroyed LOCK_thd_data mutex. Another problem is that 'delete thd' in backup threads is called outside of LOCK_thread_count lock/unlock which can cause the crash as well. The fix is to protect 'delete thd' call with LOCK_thread_count and remove deleted thd from thread list. @ sql/backup/be_thread.cc protect 'delete thd' call with LOCK_thread_count @ sql/sql_class.cc added deleted thd removal from thread list @ sql/sql_list.h added is_linked() method @ storage/myisam/myisam_backup_engine.cc protect 'delete thd' call with LOCK_thread_count
[9 Dec 2009 14:58]
Rafal Somla
Good to push. Note however that myisam_backup_egnine.cc is also fixed in BUG#49342.
[10 Dec 2009 9:55]
Bugs System
A patch for this bug has been committed. After review, it may be pushed to the relevant source trees for release in the next version. You can access the patch from: http://lists.mysql.com/commits/93466 2906 Sergey Glukhov 2009-12-10 Bug#43357 Server crash when selecting from INFORMATION_SCHEMA Server crashes in mysqld_list_processes() function on pthread_mutex_unlock(&tmp->LOCK_thd_data) call. It happens because threads after its delete are still visible in thread list with destroyed LOCK_thd_data mutex. The fix is to remove deleted thd from thread list. @ sql/sql_class.cc added deleted thd removal from thread list @ sql/sql_list.h added is_linked() method
[15 Dec 2009 17:04]
Bugs System
A patch for this bug has been committed. After review, it may be pushed to the relevant source trees for release in the next version. You can access the patch from: http://lists.mysql.com/commits/94213 2907 Chuck Bell 2009-12-15 BUG#43357 Server crash when selecting from INFORMATION_SCHEMA Server crashes in mysqld_list_processes() function on pthread_mutex_unlock(&tmp->LOCK_thd_data) call. It happens because threads after its delete are still visible in thread list with destroyed LOCK_thd_data mutex. The fix is to remove deleted thd from thread list. @ sql/sql_class.cc added deleted thd removal from thread list @ sql/sql_list.h added is_linked() method
[20 Feb 2010 9:18]
Bugs System
Pushed into 6.0.14-alpha (revid:ingo.struewing@sun.com-20100218152520-s4v1ld76bif06eqn) (version source revid:ingo.struewing@sun.com-20100119103538-wtp5alpz4p2jayl5) (merge vers: 6.0.14-alpha) (pib:16)
[25 Feb 2010 0:19]
Paul DuBois
Noted in 6.0.14 changelog. MySQL Backup sometimes crashed when selecting information from INFORMATION_SCHEMA.