Bug #48425 | concurrency update with select...union may crash the server | ||
---|---|---|---|
Submitted: | 30 Oct 2009 5:08 | Modified: | 31 Jan 2010 9:50 |
Reporter: | Xindong Su | Email Updates: | |
Status: | No Feedback | Impact on me: | |
Category: | MySQL Server | Severity: | S1 (Critical) |
Version: | 5.0.85,5.0.87 | OS: | Linux (CentOS 5.2 x86_64) |
Assigned to: | CPU Architecture: | Any | |
Tags: | update select union crash |
[30 Oct 2009 5:08]
Xindong Su
[30 Oct 2009 5:24]
Valeriy Kravchuk
Thank you for the problem report. Please, send the results of: show table status like 'dd'\G select count(distinct DD_TableName) from dd;
[30 Oct 2009 6:37]
Xindong Su
mysql> show table status like 'dd'; +------+------------+---------+------------+------+----------------+-------------+-----------------+--------------+-----------+----------------+-------------+-------------+------------+-------------------+----------+----------------+-----------------------+ | Name | Engine | Version | Row_format | Rows | Avg_row_length | Data_length | Max_data_length | Index_length | Data_free | Auto_increment | Create_time | Update_time | Check_time | Collation | Checksum | Create_options | Comment | +------+------------+---------+------------+------+----------------+-------------+-----------------+--------------+-----------+----------------+-------------+-------------+------------+-------------------+----------+----------------+-----------------------+ | dd | ndbcluster | 10 | Dynamic | 4028 | 1092 | 5373952 | 0 | 0 | 0 | 6460 | NULL | NULL | NULL | latin1_swedish_ci | NULL | | number_of_replicas: 2 | +------+------------+---------+------------+------+----------------+-------------+-----------------+--------------+-----------+----------------+-------------+-------------+------------+-------------------+----------+----------------+-----------------------+ 1 row in set (0.01 sec) mysql> select count(distinct DD_Tablename) from dd; +------------------------------+ | count(distinct DD_Tablename) | +------------------------------+ | 247 | +------------------------------+ 1 row in set (0.02 sec) I have tried different storage engines: MyISAM and NDB. But I have not ever tried InnoDB.
[30 Oct 2009 6:38]
Xindong Su
NDBD did not crash when mysqld crashed. So I think this is a bug relate to mysqld.
[30 Oct 2009 16:49]
Valeriy Kravchuk
Can you upload a dump of dd table data (as private file, if you want). I can not repeat with my dummy data generated.
[31 Oct 2009 14:38]
Xindong Su
Table dump had been uploaded. Please see "Files" for it. This bug is not always happened. Sometimes the crash log contains no SQL statement, like this: 091030 16:59:54 - mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. key_buffer_size=16777216 read_buffer_size=524288 max_used_connections=439 max_connections=1000 threads_connected=419 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_connections = 1552384 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. thd=0x2aaabd034540 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... Cannot determine thread, fp=0x43257fd0, backtrace may not be correct. Stack range sanity check OK, backtrace follows: (nil) Stack trace seems successful - bottom reached Please read http://dev.mysql.com/doc/mysql/en/using-stack-trace.html and follow instructions on how to resolve the stack trace. Resolved stack trace is much more helpful in diagnosing the problem, so please do resolve it Trying to get some variables. Some pointers may be invalid and cause the dump to abort... thd->query at 0x2aaab4959e70 is invalid pointer thd->thread_id=376683 The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. Number of processes running now: 0 091030 16:59:54 mysqld restarted 091030 16:59:58 [Note] /usr/sbin/mysqld: ready for connections. Version: '5.0.87-community-log' socket: '/var/lib/mysql/mysql.sock' port: 3306 MySQL Community Edition (GPL) How to make the error log more detailed? I am using rpm packages and I have installed the debug info package but how to use it?
[31 Oct 2009 15:19]
Valeriy Kravchuk
Can not repeat this crash with your dd table dump and recent 5.0.88 from bzr. Please, send the results of: free df -k Linux commands from your system.
[1 Nov 2009 7:55]
Xindong Su
[root@mysql1 ~]# free total used free shared buffers cached Mem: 16430804 16214524 216280 0 709908 13617072 -/+ buffers/cache: 1887544 14543260 Swap: 2096440 200 2096240 [root@mysql1 ~]# df -k 文件系统 1K-块 已用 可用 已用% 挂载点 /dev/sda8 4061540 562024 3289872 15% / /dev/sda7 4061540 73928 3777968 2% /tmp /dev/sda6 20315812 5226044 14041128 28% /home /dev/sda5 20315812 3995248 15271924 21% /usr /dev/sda3 203145268 31503988 161155656 17% /var /dev/sda2 256665 23516 219896 10% /boot tmpfs 8215400 0 8215400 0% /dev/shm /home/client/CentOS_5.2_Final.iso 4493152 4493152 0 100% /mnt [root@mysql1 ~]#
[2 Nov 2009 7:26]
Xindong Su
After turnning on the general log, I found that this bug is not so simple. Here is the latest crash log and the general log: Crash log: 091102 15:16:50 - mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. key_buffer_size=16777216 read_buffer_size=262144 max_used_connections=390 max_connections=1000 threads_connected=363 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_connections = 784384 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. thd=0x2aaab01da900 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... Cannot determine thread, fp=0x42e18fd0, backtrace may not be correct. Stack range sanity check OK, backtrace follows: (nil) Stack trace seems successful - bottom reached Please read http://dev.mysql.com/doc/mysql/en/using-stack-trace.html and follow instructions on how to resolve the stack trace. Resolved stack trace is much more helpful in diagnosing the problem, so please do resolve it Trying to get some variables. Some pointers may be invalid and cause the dump to abort... thd->query at 0x1803ad50 = (SELECT 0 as ResultSetOrder, dd.* FROM dd WHERE DD_FieldIsHide="N" and (DD_DatabaseName="JWXT") and (DD_TableName="cyd")) UNION ALL (SELECT 1 as ResultSetOrder, dd.* FROM dd WHERE DD_FieldIsHide="N" and (DD_DatabaseName="JWXT") and (DD_TableName="cydsp")) ORDER BY ResultSetOrder ASC, DD_DatabaseName ASC, DD_TableName ASC, DD_FieldQueryOrderPriority DESC, DD_FieldNO ASC thd->thread_id=217 The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. Number of processes running now: 0 091102 15:16:50 mysqld restarted 091102 15:16:54 [Note] /usr/sbin/mysqld: ready for connections. Version: '5.0.87-community-log' socket: '/var/lib/mysql/mysql.sock' port: 3306 MySQL Community Edition (GPL) Here is the last second of the general log: 217 Query (SELECT 0 as ResultSetOrder, dd.* FROM dd WHERE DD_FieldIsHide="N" and (DD_DatabaseName="JWXT") and (DD_TableName="cyd")) UNION ALL (SELECT 1 as ResultSetOrder, dd.* FROM dd WHERE DD_FieldIsHide="N" and (DD_DatabaseName="JWXT") and (DD_TableName="cydsp")) ORDER BY ResultSetOrder, DD_DatabaseName, DD_TableName, DD_FieldNO 091102 15:16:50 10954 Query SET AUTOCOMMIT=0 10954 Query UPDATE dd SET DD_FieldQueryOrderPriority=DD_FieldQueryOrderPriority+1 WHERE DD_TableName="cyd" and FieldName="InspectionStandard" 217 Query (SELECT 0 as ResultSetOrder, dd.* FROM dd WHERE DD_FieldIsHide="N" and (DD_DatabaseName="JWXT") and (DD_TableName="cyd")) UNION ALL (SELECT 1 as ResultSetOrder, dd.* FROM dd WHERE DD_FieldIsHide="N" and (DD_DatabaseName="JWXT") and (DD_TableName="cydsp")) ORDER BY ResultSetOrder ASC, DD_DatabaseName ASC, DD_TableName ASC, DD_FieldQueryOrderPriority DESC, DD_FieldNO ASC Maybe the update operation to the table 'dd' is the real cause of the crash. Please check it out.
[2 Nov 2009 7:29]
Xindong Su
last 2 seconds of the general log when mysqld crashed.
Attachment: last_2_seconds_general_log.zip (application/zip, text), 2.14 KiB.
[3 Nov 2009 5:00]
Xindong Su
After researching to the general log, I realized that this bug may be caused by the concurrency UPDATE and SELECT...UNION operation by different thread. General log Example 1: 217 Query (SELECT 0 as ResultSetOrder, dd.* FROM dd WHERE DD_FieldIsHide="N" and (DD_DatabaseName="JWXT") and (DD_TableName="cyd")) UNION ALL (SELECT 1 as ResultSetOrder, dd.* FROM dd WHERE DD_FieldIsHide="N" and (DD_DatabaseName="JWXT") and (DD_TableName="cydsp")) ORDER BY ResultSetOrder, DD_DatabaseName, DD_TableName, DD_FieldNO 10954 Query SET AUTOCOMMIT=0 10954 Query UPDATE dd SET DD_FieldQueryOrderPriority=DD_FieldQueryOrderPriority+1 WHERE DD_TableName="cyd" and FieldName="InspectionStandard" 217 Query (SELECT 0 as ResultSetOrder, dd.* FROM dd WHERE DD_FieldIsHide="N" and (DD_DatabaseName="JWXT") and (DD_TableName="cyd")) UNION ALL (SELECT 1 as ResultSetOrder, dd.* FROM dd WHERE DD_FieldIsHide="N" and (DD_DatabaseName="JWXT") and (DD_TableName="cydsp")) ORDER BY ResultSetOrder ASC, DD_DatabaseName ASC, DD_TableName ASC, DD_FieldQueryOrderPriority DESC, DD_FieldNO ASC mysqld crashed after this statement. General log example 2: 18283 Query SET AUTOCOMMIT=0 18283 Query UPDATE dd SET DD_FieldQueryOrderPriority=DD_FieldQueryOrderPriority+1 WHERE DD_TableName="cyd" and FieldName="SamplingSheetNO" 148384 Query (SELECT 0 as ResultSetOrder, dd.* FROM dd WHERE DD_FieldIsHide="N" and (DD_DatabaseName="JWXT") and (DD_TableName="cyd")) UNION ALL (SELECT 1 as ResultSetOrder, dd.* FROM dd WHERE DD_FieldIsHide="N" and (DD_DatabaseName="JWXT") and (DD_TableName="cydsp")) UNION ALL (SELECT 2 as ResultSetOrder, dd.* FROM dd WHERE DD_FieldIsHide="N" and (DD_DatabaseName="JWXT") and (DD_TableName="fc")) ORDER BY ResultSetOrder, DD_DatabaseName, DD_TableName, DD_FieldNO 18283 Query COMMIT 18283 Query SET AUTOCOMMIT=1 mysqld crashed after this statement. So I modified my program, disabled the UPDATE operation to the table. The mysqld doesn't crashed since then.
[17 Dec 2009 14:33]
Susanne Ebrecht
I am not able to repeat this with MySQL 5.1.
[30 Dec 2009 22:56]
MySQL Verification Team
Are you able to provide a repeatable test case?. Thanks in advance.
[31 Dec 2009 9:44]
Xindong Su
I will try.
[1 Feb 2010 0:00]
Bugs System
No feedback was provided for this bug for over a month, so it is being suspended automatically. If you are able to provide the information that was originally requested, please do so and change the status of the bug back to "Open".