| Bug #50557 | checksum table crashes server when used in performance_schema | ||
|---|---|---|---|
| Submitted: | 22 Jan 2010 20:01 | Modified: | 4 Nov 2010 1:26 |
| Reporter: | Shane Bester (Platinum Quality Contributor) | Email Updates: | |
| Status: | Closed | Impact on me: | |
| Category: | MySQL Server: Performance Schema | Severity: | S1 (Critical) |
| Version: | 5.5.3,5.5.99-m3-debug, 5.6.0-debug-m4 | OS: | Any |
| Assigned to: | Marc ALFF | CPU Architecture: | Any |
| Tags: | checksum, valgrind | ||
[22 Jan 2010 20:01]
Shane Bester
[22 Jan 2010 23:45]
Marc ALFF
I can not reproduce this in mysql-next-mr Script: select version(); checksum table `events_waits_current`; checksum table `events_waits_history`; use performance_schema; checksum table `EVENTS_WAITS_CURRENT`; checksum table `EVENTS_WAITS_HISTORY`; Result: select version(); version() 5.5.99-m3-debug-log checksum table `events_waits_current`; Table Checksum test.events_waits_current NULL Warnings: Error 1146 Table 'test.events_waits_current' doesn't exist checksum table `events_waits_history`; Table Checksum test.events_waits_history NULL Warnings: Error 1146 Table 'test.events_waits_history' doesn't exist use performance_schema; checksum table `EVENTS_WAITS_CURRENT`; Table Checksum performance_schema.EVENTS_WAITS_CURRENT 1022703524 checksum table `EVENTS_WAITS_HISTORY`; Table Checksum performance_schema.EVENTS_WAITS_HISTORY 220092765
[23 Jan 2010 9:25]
MySQL Verification Team
Hi Marc! Sorry for the confusion, but it's still a bug. This is a windows only bug I found out now. I build on linux and got no crash. Windows crashes here in crc32_little():
while (len >= 32) {
DOLIT32; <----
len -= 32;
}
[23 Jan 2010 9:34]
MySQL Verification Team
linux has a problem too (and not bug #49465 ). run under valgrind to see it! Use of uninitialised value of size 4 at: crc32 (in /usr/lib/libz.so.1.2.3) by: my_checksum (checksum.c:39) by: mysql_checksum_table(sql_table.cc:7921) by: mysql_execute_command(THD*) (sql_parse.cc:2845) by: mysql_parse(sql_parse.cc:5989) by: dispatch_command(sql_parse.cc:1140) by: do_command(THD*) (sql_parse.cc:810) by: do_handle_one_connection(THD*) (sql_connect.cc:1172) by: handle_one_connection (sql_connect.cc:1112) by: pfs_spawn_thread(void*) (pfs.cc:1011) by: start_thread (in /lib/libpthread-2.5.so) by: clone (in /lib/libc-2.5.so) Notice the checksum table reports a different checksum each time even when the data didn't change ?
[25 Jan 2010 9:14]
Sveta Smirnova
Thank you for the report. Valgrind error verified as described. ==25347== Address 0x8ffbdf4 is 6,132 bytes inside a block of size 8,084 free'd ==25347== at 0x4A062ED: free (vg_replace_malloc.c:323) ==25347== by 0xB29284: _myfree (safemalloc.c:314) ==25347== by 0xB285FF: free_root (my_alloc.c:355) ==25347== by 0x6F430C: open_tables(THD*, TABLE_LIST**, unsigned int*, unsigned int) (sql_base.cc:4798) ==25347== by 0x6F4947: open_and_lock_tables_derived(THD*, TABLE_LIST*, bool) (sql_base.cc:5027) ==25347== by 0x6A9BAB: simple_open_n_lock_tables(THD*, TABLE_LIST*) (mysql_priv.h:1570) ==25347== by 0xB0B4A2: PFS_engine_table_share::check_one_table(THD*) (pfs_engine_table.cc:129) ==25347== by 0xB0B2F7: PFS_engine_table_share::check_all_tables(THD*) (pfs_engine_table.cc:83) ==25347== by 0xB19726: check_performance_schema() (pfs_check.cc:57) ==25347== by 0x68C059: main (mysqld.cc:4792)
[29 Jan 2010 20:06]
Marc ALFF
There is too much confusion about this bug report. If the reported bug is that checksum on performance schema tables is not constant, this is the expected result: the content of tables like EVENTS_WAITS_HISTORY changes all the time. If the reported bug is that checksum crashes on a performance schema table, then please also test checksum on a non performance schema table, and indicate clearly in the bug report the results. If the crash is reproduced without the performance schema, then change the bug category. If the bug reported is about a valgrind error, then please indicate which test case was run with valgrind, and on which platform. I could not reproduce the crash, or any valgrind error, on my machine (linux). Changing the status to need feedback. Thanks, -- Marc
[1 Feb 2010 6:14]
Sveta Smirnova
Marc, here is the answer on some of your questions. About Valgrind error. Test from your comment. $cat t/bug50557.test select version(); checksum table `events_waits_current`; checksum table `events_waits_history`; use performance_schema; checksum table `EVENTS_WAITS_CURRENT`; checksum table `EVENTS_WAITS_HISTORY`; $cat t/bug50557-master.opt --performance-schema=1
[1 Mar 2010 0:00]
Bugs System
No feedback was provided for this bug for over a month, so it is being suspended automatically. If you are able to provide the information that was originally requested, please do so and change the status of the bug back to "Open".
[17 Mar 2010 8:48]
Bugs System
A patch for this bug has been committed. After review, it may be pushed to the relevant source trees for release in the next version. You can access the patch from: http://lists.mysql.com/commits/103543 3123 Marc Alff 2010-03-17 Bug#50557 checksum table crashes server when used in performance_schema This bug could not be reproduced. Adding the test case to the test suite for coverage.
[17 Mar 2010 14:06]
Kristofer Pettersson
Attempted to reproduce the error on Ubuntu but were unable to: All heap blocks were freed -- no leaks are possible. I'm unable to test this in windows and Shane claims it is a windows specific bug however the test case looks good for code coverage and I approve of it. I need to get a working windows environment to test this further. Marc, did you test this on windows?
[18 Mar 2010 6:39]
MySQL Verification Team
hi! there's no memory leak with this bug. The bug is invalid reads of memory, which is reproducible very very easily. Version: '5.6.99-m4-debug' socket: '' port: 3306 Source distribution 100318 8:38:40 - mysqld got exception 0xc0000005 ; mysqld.exe!crc32_little()[crc32.c:279] mysqld.exe!crc32()[crc32.c:237] mysqld.exe!my_checksum()[checksum.c:39] mysqld.exe!mysql_checksum_table()[sql_table.cc:7964] mysqld.exe!mysql_execute_command()[sql_parse.cc:2894] mysqld.exe!mysql_parse()[sql_parse.cc:5735] mysqld.exe!dispatch_command()[sql_parse.cc:1024] mysqld.exe!do_command()[sql_parse.cc:710] mysqld.exe!do_handle_one_connection()[sql_connect.cc:1174] mysqld.exe!handle_one_connection()[sql_connect.cc:1113] mysqld.exe!pfs_spawn_thread()[pfs.cc:1011] mysqld.exe!pthread_start()[my_winthread.c:61] mysqld.exe!_callthreadstartex()[threadex.c:348] mysqld.exe!_threadstartex()[threadex.c:331] kernel32.dll!FlsSetValue() et some variables. rs may be invalid and cause the dump to abort... at 13660528=checksum table events_waits_current extended My build: E:\builds\bzr\mysql-next-mr-bugfixing\sql\debug>bzr log|head -n 20 ------------------------------------------------------------ revno: 3128 committer: Vladislav Vaintroub <vvaintroub@mysql.com> branch nick: mysql-next-mr-bugfixing timestamp: Wed 2010-03-17 20:31:04 +0100 message:
[18 Mar 2010 13:58]
Marc ALFF
Shane, I don't have access to a Windows machine so I can not reproduce this. If you can reproduce it, then please also try a checksum on a regular table, and document the results, so that this bug can be narrowed and properly qualified. I asked this the last time when asking for feedback because of the general confusion, and still got no answers. Changing to need feedback again. Thanks, -- Marc
[18 Apr 2010 23:00]
Bugs System
No feedback was provided for this bug for over a month, so it is being suspended automatically. If you are able to provide the information that was originally requested, please do so and change the status of the bug back to "Open".
[19 Apr 2010 15:03]
MySQL Verification Team
Version: '5.5.3-m3-community' socket: '' port: 3306 MySQL Community Server (GPL) 100419 17:03:35 - mysqld got exception 0xc0000005 ; 140340AF0 crc32_little()[crc32.c:279] 1400E22B5 mysql_checksum_table()[sql_table.cc:7964] 14006AA7F mysql_execute_command()[sql_parse.cc:2894] 14006D79D mysql_parse()[sql_parse.cc:5740] 14006E3C2 dispatch_command()[sql_parse.cc:1026] 14006F24C do_command()[sql_parse.cc:710] 140092041 do_handle_one_connection()[sql_connect.cc:1174] 1400920F4 handle_one_connection()[sql_connect.cc:1115] 140317B39 pfs_spawn_thread()[pfs.cc:1014] 1403210AE pthread_start()[my_winthread.c:63] 1403A8207 _callthreadstartex()[threadex.c:348] 1403A82BB _threadstartex()[threadex.c:326] 077D6B6CA BaseThreadStart() to get some variables. inters may be invalid and cause the dump to abort... ery at 000000000FEF8160=/*extreme*/checksum table `EVENTS_WAITS_CURRENT`
[4 Sep 2010 8:33]
MySQL Verification Team
still repeatable on 5.6.0-debug-m4 Version: '5.6.0-m4-debug-log' socket: '' port: 3306 MySQL Community Server - Debug (GPL) 100904 10:32:10 - mysqld got exception 0xc0000005 ; crc32_little()[crc32.c:279] crc32()[crc32.c:237] my_checksum()[checksum.c:34] mysql_checksum_table()[sql_table.cc:8167] mysql_execute_command()[sql_parse.cc:2942] mysql_parse()[sql_parse.cc:5828] dispatch_command()[sql_parse.cc:1132] do_command()[sql_parse.cc:801] do_handle_one_connection()[sql_connect.cc:1191] handle_one_connection()[sql_connect.cc:1131] pfs_spawn_thread()[pfs.cc:1063] pthread_start()[my_winthread.c:62] _callthreadstartex()[threadex.c:348] _threadstartex()[threadex.c:331] BaseThreadStart() Some pointers may be invalid and cause the dump to abort... thd->query at 000000000DE320B0=checksum table performance_schema.events_waits_current extended
[6 Sep 2010 20:02]
Sveta Smirnova
Crash sporadically repeatable on Windows. I run test 2 times.
[9 Sep 2010 12:54]
Elena Stepanova
Got the same crash in system tests on trunk-release (5.6.1-m4-log).
[15 Sep 2010 16:30]
Marc ALFF
Possibly related to bug#56761
[16 Sep 2010 21:36]
Marc ALFF
When building: - mysql-5.5-bugfixing, - revision number 3207 (which contains the fix for bug#56761), the following test: - suite=perfschema, checksum runs under valgrind with no problem on my development machine (Linux, 64 bit). Waiting for more tests results from PB2. If this tests still fails, *PLEASE* provide technical details.
[16 Sep 2010 21:54]
Marc ALFF
Likewise for: - mysql-trunk-bugfixing, revno 3267 - mysql-next-mr-bugfixing, revno 3289 the checksum test, that cover all performance schema tables, and both the extended and non extended case, do pass under valgrind. If a core happen on any machine, please try to determine the following information: - what column is the checksum crashing on ? - what's the column type ? - what's the column value ? - what's the column length ? for strings, does the content look like a likely string, or is it garbage ? Thanks
[17 Sep 2010 19:03]
Bugs System
A patch for this bug has been committed. After review, it may be pushed to the relevant source trees for release in the next version. You can access the patch from: http://lists.mysql.com/commits/118520 3210 Marc Alff 2010-09-17 Bug#50557 checksum table crashes server when used in performance_schema CHECKSUM TABLE for performance schema tables could cause uninitialized memory reads. The root cause is a design flaw in the implementation of mysql_checksum_table(), which do not honor null fields. However, fixing this bug in CHECKSUM TABLE is risky, as it can cause the checksum value to change. This fix implements a work around, to systematically reset fields values even for null fields, so that the field memory representation is always initialized with a known value.
[17 Sep 2010 19:10]
Marc ALFF
Shane, Sveta, Elena, Jon Olav, I have never been able to reproduce the valgrind UMR failure or the crash itself, no matter which tree (5.5, trunk, next-mr) was used, and no matter which compiling option were used. Trust me, I have spent a lot of time on this already. However, by analysis of the source code of mysql_check_table(), I think there is a possible root cause for the UMR failure itself. Could everyone of you, since you were able to verify the problem at some point, please try to apply the last patch to mysql-5.5-bugfixing and document the results ? Your help to narrow this down is appreciated. Thanks, -- Marc
[17 Sep 2010 19:42]
Sveta Smirnova
Patch fixes problem for me.
[20 Sep 2010 6:14]
Jon Olav Hauglid
The patch fixes the problem for me as well.
[20 Sep 2010 18:09]
Christopher Powers
Patch approved.
[20 Sep 2010 23:22]
Marc ALFF
Pushed into: - mysql-5.5-bugfixing - mysql-trunk-bugfixing - mysql-next-mr-bugfixing
[2 Oct 2010 18:13]
Bugs System
Pushed into mysql-trunk 5.6.1-m4 (revid:alexander.nozdrin@oracle.com-20101002180948-852x1cuv7c6i85ea) (version source revid:alexander.nozdrin@oracle.com-20101002180857-an32jpuwzemsp4f2) (merge vers: 5.6.1-m4) (pib:21)
[2 Oct 2010 18:14]
Bugs System
Pushed into mysql-next-mr (revid:alexander.nozdrin@oracle.com-20101002181053-6iotvl26uurcoryp) (version source revid:alexander.nozdrin@oracle.com-20101002180917-h0n62akupm3z20nt) (pib:21)
[2 Oct 2010 18:17]
Bugs System
Pushed into mysql-5.5 5.5.7-rc (revid:alexander.nozdrin@oracle.com-20101002180831-590ka2tuit9qoxbb) (version source revid:alexander.nozdrin@oracle.com-20101002180831-590ka2tuit9qoxbb) (merge vers: 5.5.7-rc) (pib:21)
[4 Nov 2010 1:26]
Paul DuBois
Noted in 5.5.7, 5.6.1 changelogs. CHECKSUM TABLE for Performance Schema tables could cause a server crash due to uninitialized memory reads.
