Bug #50557 checksum table crashes server when used in performance_schema
Submitted: 22 Jan 2010 20:01 Modified: 4 Nov 2010 1:26
Reporter: Shane Bester (Platinum Quality Contributor) Email Updates:
Status: Closed Impact on me:
None 
Category:MySQL Server: Performance Schema Severity:S1 (Critical)
Version:5.5.3,5.5.99-m3-debug, 5.6.0-debug-m4 OS:Any
Assigned to: Marc Alff
Tags: checksum, valgrind
Triage: Triaged: D1 (Critical)

[22 Jan 2010 20:01] Shane Bester
Description:
See related bug #50555
I branched mysql-next-mr-bugfixing and built debug server that crashes

mysqld.exe!crc32_little()[crc32.c:279]
mysqld.exe!crc32()[crc32.c:237]
mysqld.exe!my_checksum()[checksum.c:39]
mysqld.exe!mysql_checksum_table()[sql_table.cc:7917]
mysqld.exe!mysql_execute_command()[sql_parse.cc:2845]
mysqld.exe!mysql_parse()[sql_parse.cc:5989]
mysqld.exe!dispatch_command()[sql_parse.cc:1140]
mysqld.exe!do_command()[sql_parse.cc:810]
mysqld.exe!do_handle_one_connection()[sql_connect.cc:1172]
mysqld.exe!handle_one_connection()[sql_connect.cc:1112]
mysqld.exe!pfs_spawn_thread()[pfs.cc:1011]
mysqld.exe!pthread_start()[my_winthread.c:61]
mysqld.exe!_callthreadstartex()[threadex.c:348]
mysqld.exe!_threadstartex()[threadex.c:331]
kernel32.dll!FlsSetValue()

Exact revision was this:
[sbester@box1 mysql-next-mr-bugfixing]$ bzr log |head -n 20
------------------------------------------------------------
revno: 2972
committer: Alexander Nozdrin <alik@sun.com>
branch nick: mysql-next-mr-bugfixing
timestamp: Fri 2010-01-22 11:20:13 +0300
message:
  Auto-merge from mysql-next-mr.

How to repeat:
start mysqld with --performance-schema=1
run this:

checksum table `events_waits_history`
[22 Jan 2010 23:45] Marc Alff
I can not reproduce this in mysql-next-mr

Script:
select version();

checksum table `events_waits_current`;
checksum table `events_waits_history`;

use performance_schema;
checksum table `EVENTS_WAITS_CURRENT`;
checksum table `EVENTS_WAITS_HISTORY`;

Result:
select version();
version()
5.5.99-m3-debug-log
checksum table `events_waits_current`;
Table   Checksum
test.events_waits_current       NULL
Warnings:
Error   1146    Table 'test.events_waits_current' doesn't exist
checksum table `events_waits_history`;
Table   Checksum
test.events_waits_history       NULL
Warnings:
Error   1146    Table 'test.events_waits_history' doesn't exist
use performance_schema;
checksum table `EVENTS_WAITS_CURRENT`;
Table   Checksum
performance_schema.EVENTS_WAITS_CURRENT 1022703524
checksum table `EVENTS_WAITS_HISTORY`;
Table   Checksum
performance_schema.EVENTS_WAITS_HISTORY 220092765
[23 Jan 2010 9:34] Shane Bester
linux has a problem too (and not bug #49465 ).  run under valgrind to see it!

Use of uninitialised value of size 4                                                                  
at: crc32 (in /usr/lib/libz.so.1.2.3)                                                     
by: my_checksum (checksum.c:39)                                                          
by: mysql_checksum_table(sql_table.cc:7921)        
by: mysql_execute_command(THD*) (sql_parse.cc:2845)                                      
by: mysql_parse(sql_parse.cc:5989)       
by: dispatch_command(sql_parse.cc:1140) 
by: do_command(THD*) (sql_parse.cc:810)                                                  
by: do_handle_one_connection(THD*) (sql_connect.cc:1172)                                 
by: handle_one_connection (sql_connect.cc:1112)                                          
by: pfs_spawn_thread(void*) (pfs.cc:1011)                                                
by: start_thread (in /lib/libpthread-2.5.so)                                              
by: clone (in /lib/libc-2.5.so)                                                           

Notice the checksum table reports a different checksum each time even when the data didn't change ?
[29 Jan 2010 20:06] Marc Alff
There is too much confusion about this bug report.

If the reported bug is that checksum on performance schema tables is not constant, this is the expected result: the content of tables like EVENTS_WAITS_HISTORY changes all the time.

If the reported bug is that checksum crashes on a performance schema table, then please also test checksum on a non performance schema table, and indicate clearly in the bug report the results. If the crash is reproduced without the performance schema, then change the bug category.

If the bug reported is about a valgrind error, then please indicate which test case was run with valgrind, and on which platform.

I could not reproduce the crash, or any valgrind error, on my machine (linux).

Changing the status to need feedback.

Thanks,
-- Marc
[1 Feb 2010 6:14] Sveta Smirnova
Marc,

here is the answer on some of your questions.

About Valgrind error. Test from your comment.

$cat t/bug50557.test 
select version();

checksum table `events_waits_current`;
checksum table `events_waits_history`;

use performance_schema;
checksum table `EVENTS_WAITS_CURRENT`;
checksum table `EVENTS_WAITS_HISTORY`;

$cat t/bug50557-master.opt 
--performance-schema=1
[1 Mar 2010 0:00] Bugs System
No feedback was provided for this bug for over a month, so it is
being suspended automatically. If you are able to provide the
information that was originally requested, please do so and change
the status of the bug back to "Open".
[17 Mar 2010 8:48] Bugs System
A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/103543

3123 Marc Alff	2010-03-17
      Bug#50557 checksum table crashes server when used in performance_schema
      
      This bug could not be reproduced.
      Adding the test case to the test suite for coverage.
[17 Mar 2010 14:06] Kristofer Pettersson
Attempted to reproduce the error on Ubuntu but were unable to: All heap blocks were freed -- no leaks are possible.

I'm unable to test this in windows and Shane claims it is a windows specific bug however the test case looks good for code coverage and I approve of it.

I need to get a working windows environment to test this further.

Marc, did you test this on windows?
[18 Mar 2010 6:39] Shane Bester
hi!  there's no memory leak with this bug.  The bug is invalid reads of memory, which is reproducible very very easily.

Version: '5.6.99-m4-debug'  socket: ''  port: 3306  Source distribution
100318  8:38:40 - mysqld got exception 0xc0000005 ;

 mysqld.exe!crc32_little()[crc32.c:279]
 mysqld.exe!crc32()[crc32.c:237]
 mysqld.exe!my_checksum()[checksum.c:39]
 mysqld.exe!mysql_checksum_table()[sql_table.cc:7964]
 mysqld.exe!mysql_execute_command()[sql_parse.cc:2894]
 mysqld.exe!mysql_parse()[sql_parse.cc:5735]
 mysqld.exe!dispatch_command()[sql_parse.cc:1024]
 mysqld.exe!do_command()[sql_parse.cc:710]
 mysqld.exe!do_handle_one_connection()[sql_connect.cc:1174]
 mysqld.exe!handle_one_connection()[sql_connect.cc:1113]
 mysqld.exe!pfs_spawn_thread()[pfs.cc:1011]
 mysqld.exe!pthread_start()[my_winthread.c:61]
 mysqld.exe!_callthreadstartex()[threadex.c:348]
 mysqld.exe!_threadstartex()[threadex.c:331]
 kernel32.dll!FlsSetValue()
et some variables.
rs may be invalid and cause the dump to abort...
at 13660528=checksum table events_waits_current extended

My build:
E:\builds\bzr\mysql-next-mr-bugfixing\sql\debug>bzr log|head -n 20
------------------------------------------------------------
revno: 3128
committer: Vladislav Vaintroub <vvaintroub@mysql.com>
branch nick: mysql-next-mr-bugfixing
timestamp: Wed 2010-03-17 20:31:04 +0100
message:
[18 Mar 2010 13:58] Marc Alff
Shane,

I don't have access to a Windows machine so I can not reproduce this.

If you can reproduce it, then please also try a checksum on a regular table, and document the results, so that this bug can be narrowed and properly qualified.

I asked this the last time when asking for feedback because of the general confusion, and still got no answers.

Changing to need feedback again.

Thanks,
-- Marc
[18 Apr 2010 23:00] Bugs System
No feedback was provided for this bug for over a month, so it is
being suspended automatically. If you are able to provide the
information that was originally requested, please do so and change
the status of the bug back to "Open".
[19 Apr 2010 15:03] Shane Bester
Version: '5.5.3-m3-community'  socket: ''  port: 3306  MySQL Community Server (GPL)
100419 17:03:35 - mysqld got exception 0xc0000005 ;

140340AF0    crc32_little()[crc32.c:279]
1400E22B5    mysql_checksum_table()[sql_table.cc:7964]
14006AA7F    mysql_execute_command()[sql_parse.cc:2894]
14006D79D    mysql_parse()[sql_parse.cc:5740]
14006E3C2    dispatch_command()[sql_parse.cc:1026]
14006F24C    do_command()[sql_parse.cc:710]
140092041    do_handle_one_connection()[sql_connect.cc:1174]
1400920F4    handle_one_connection()[sql_connect.cc:1115]
140317B39    pfs_spawn_thread()[pfs.cc:1014]
1403210AE    pthread_start()[my_winthread.c:63]
1403A8207    _callthreadstartex()[threadex.c:348]
1403A82BB    _threadstartex()[threadex.c:326]
077D6B6CA    BaseThreadStart()
to get some variables.
inters may be invalid and cause the dump to abort...
ery at 000000000FEF8160=/*extreme*/checksum table `EVENTS_WAITS_CURRENT`
[4 Sep 2010 8:33] Shane Bester
still repeatable on 5.6.0-debug-m4

Version: '5.6.0-m4-debug-log'  socket: ''  port: 3306  MySQL Community Server - Debug (GPL)
100904 10:32:10 - mysqld got exception 0xc0000005 ;
 crc32_little()[crc32.c:279]
 crc32()[crc32.c:237]
 my_checksum()[checksum.c:34]
 mysql_checksum_table()[sql_table.cc:8167]
 mysql_execute_command()[sql_parse.cc:2942]
 mysql_parse()[sql_parse.cc:5828]
 dispatch_command()[sql_parse.cc:1132]
 do_command()[sql_parse.cc:801]
 do_handle_one_connection()[sql_connect.cc:1191]
 handle_one_connection()[sql_connect.cc:1131]
 pfs_spawn_thread()[pfs.cc:1063]
 pthread_start()[my_winthread.c:62]
 _callthreadstartex()[threadex.c:348]
 _threadstartex()[threadex.c:331]
 BaseThreadStart()

Some pointers may be invalid and cause the dump to abort...
thd->query at 000000000DE320B0=checksum table performance_schema.events_waits_current extended
[6 Sep 2010 20:02] Sveta Smirnova
Crash sporadically repeatable on Windows. I run test 2 times.
[9 Sep 2010 12:54] Elena Stepanova
Got the same crash in system tests on trunk-release (5.6.1-m4-log).
[15 Sep 2010 16:30] Marc Alff
Possibly related to bug#56761
[16 Sep 2010 21:36] Marc Alff
When building:
- mysql-5.5-bugfixing,
- revision number 3207 (which contains the fix for bug#56761),

the following test:
- suite=perfschema, checksum

runs under valgrind with no problem on my development machine (Linux, 64 bit).

Waiting for more tests results from PB2.

If this tests still fails, *PLEASE* provide technical details.
[16 Sep 2010 21:54] Marc Alff
Likewise for:

- mysql-trunk-bugfixing, revno 3267
- mysql-next-mr-bugfixing, revno 3289

the checksum test, that cover all performance schema tables,
and both the extended and non extended case, do pass under valgrind.

If a core happen on any machine, please try to determine the following information:
- what column is the checksum crashing on ?
- what's the column type ?
- what's the column value ?
- what's the column length ?

for strings, does the content look like a likely string, or is it garbage ?

Thanks
[17 Sep 2010 19:03] Bugs System
A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/118520

3210 Marc Alff	2010-09-17
      Bug#50557 checksum table crashes server when used in performance_schema
      
      CHECKSUM TABLE for performance schema tables could cause uninitialized
      memory reads.
      
      The root cause is a design flaw in the implementation of
      mysql_checksum_table(), which do not honor null fields.
      
      However, fixing this bug in CHECKSUM TABLE is risky, as it can cause the
      checksum value to change.
      
      This fix implements a work around, to systematically reset fields values
      even for null fields, so that the field memory representation is always
      initialized with a known value.
[17 Sep 2010 19:10] Marc Alff
Shane, Sveta, Elena, Jon Olav,

I have never been able to reproduce the valgrind UMR failure or the crash itself,
no matter which tree (5.5, trunk, next-mr) was used, and no matter which compiling option were used.

Trust me, I have spent a lot of time on this already.

However, by analysis of the source code of mysql_check_table(), I think there is a possible root cause for the UMR failure itself.

Could everyone of you, since you were able to verify the problem at some point, please try to apply the last patch to mysql-5.5-bugfixing and document the results ?

Your help to narrow this down is appreciated.

Thanks,
-- Marc
[17 Sep 2010 19:42] Sveta Smirnova
Patch fixes problem for me.
[20 Sep 2010 6:14] Jon Olav Hauglid
The patch fixes the problem for me as well.
[20 Sep 2010 18:09] Christopher Powers
Patch approved.
[20 Sep 2010 23:22] Marc Alff
Pushed into:
- mysql-5.5-bugfixing
- mysql-trunk-bugfixing
- mysql-next-mr-bugfixing
[2 Oct 2010 18:13] Bugs System
Pushed into mysql-trunk 5.6.1-m4 (revid:alexander.nozdrin@oracle.com-20101002180948-852x1cuv7c6i85ea) (version source revid:alexander.nozdrin@oracle.com-20101002180857-an32jpuwzemsp4f2) (merge vers: 5.6.1-m4) (pib:21)
[2 Oct 2010 18:14] Bugs System
Pushed into mysql-next-mr (revid:alexander.nozdrin@oracle.com-20101002181053-6iotvl26uurcoryp) (version source revid:alexander.nozdrin@oracle.com-20101002180917-h0n62akupm3z20nt) (pib:21)
[2 Oct 2010 18:17] Bugs System
Pushed into mysql-5.5 5.5.7-rc (revid:alexander.nozdrin@oracle.com-20101002180831-590ka2tuit9qoxbb) (version source revid:alexander.nozdrin@oracle.com-20101002180831-590ka2tuit9qoxbb) (merge vers: 5.5.7-rc) (pib:21)
[4 Nov 2010 1:26] Paul Dubois
Noted in 5.5.7, 5.6.1 changelogs.

CHECKSUM TABLE for Performance Schema tables could cause a server
crash due to uninitialized memory reads.