Bug #56761 Segfault on CHECKSUM TABLE performance_schema.EVENTS_WAITS_HISTORY EXTENDED
Submitted: 13 Sep 2010 22:27 Modified: 4 Nov 2010 1:21
Reporter: Elena Stepanova Email Updates:
Status: Closed Impact on me:
None 
Category:MySQL Server: Performance Schema Severity:S2 (Serious)
Version:5.6.99-m5-debug OS:Solaris
Assigned to: Marc Alff CPU Architecture:Any
Triage: Triaged: D1 (Critical)

[13 Sep 2010 22:27] Elena Stepanova
Description:
5.6.99-m5-debug-log

mysqld got signal 11 ;

=>[2] my_write_core(sig = 11), line 328 in "stacktrace.c"
  [3] handle_segfault(sig = 11), line 2507 in "mysqld.cc"
  [4] __sighndlr(0xb, 0x0, 0xffffffff72f3bd20, 0x1001d4408, 0x12, 0x0), at 0xffffffff7eec3538
  ---- called from signal handler with signal 11 (SIGSEGV) ------
  [5] memcpy(0x11868d1e8, 0xfffffffee7972e18, 0xa, 0x5, 0x73, 0x0), at 0xffffffff7effcb28
  [6] table_events_waits_common::make_row(this = 0x11868d140, thread_own_wait = true, pfs_thread = 0x1162b25e0, wait = 0x116372bf0), line 266 in "table_events_waits.cc"
  [7] table_events_waits_history::rnd_next(this = 0x11868d140), line 693 in "table_events_waits.cc"
  [8] ha_perfschema::rnd_next(this = 0x11a102e50, buf = 0x11a103240 "^P\xfe\xbf^Y"), line 274 in "ha_perfschema.cc"
  [9] handler::ha_rnd_next(this = 0x11a102e50, buf = 0x11a103240 "^P\xfe\xbf^Y"), line 2197 in "handler.cc"
  [10] mysql_checksum_table(thd = 0x12f97dd00, tables = 0x1184ea6a0, check_opt = 0x12f9804c8), line 7307 in "sql_table.cc"
  [11] mysql_execute_command(thd = 0x12f97dd00), line 2784 in "sql_parse.cc"
  [12] mysql_parse(thd = 0x12f97dd00, rawbuf = 0x1184ea540 "CHECKSUM TABLE performance_schema.EVENTS_WAITS_HISTORY EXTENDED", length = 63U, parser_state = 0xffffffff72f3f788), line 5591 in "sql_parse.cc"
  [13] dispatch_command(command = COM_QUERY, thd = 0x12f97dd00, packet = 0x12f19c6e1 "", packet_length = 63U), line 1130 in "sql_parse.cc"
  [14] do_command(thd = 0x12f97dd00), line 802 in "sql_parse.cc"
  [15] do_handle_one_connection(thd_arg = 0x12f97dd00), line 1192 in "sql_connect.cc"
  [16] handle_one_connection(arg = 0x12f97dd00), line 1131 in "sql_connect.cc"
  [17] pfs_spawn_thread(arg = 0x1312c9ff0), line 1061 in "pfs.cc"

thd->query at 1184ea540 = CHECKSUM TABLE performance_schema.EVENTS_WAITS_HISTORY EXTENDED
thd->thread_id=6532
thd->killed=NOT_KILLED

How to repeat:
No repeatable test case yet
[14 Sep 2010 0:28] Marc Alff
Elena,

I'll take this bug.

There is a failure in 
void table_events_waits_common::make_row(bool thread_own_wait,
                                         PFS_thread *pfs_thread,
                                         PFS_events_waits *wait)

inside a memcpy, which shows that the code does not do enough to sanitize the data.

"wait" is a valid pointer that may point to trashed memory,
so every access to wait->something must be sanitize.

The code does not sanitize the length before doing a memcpy, that's a problem.

This affects CHECKSUM, but also every SELECT on these performance schema tables.

Changing to verified.
[15 Sep 2010 18:21] Bugs System
A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/118318

3207 Marc Alff	2010-09-15
      Bug#56761 Segfault on CHECKSUM TABLE performance_schema.EVENTS_WAITS_HISTORY EXTENDED
      
      Before this fix, the server could crash inside a memcpy when reading data
      from the EVENTS_WAITS_CURRENT / HISTORY / HISTORY_LONG  tables.
      
      The root cause is that the length used in a memcpy could be corrupted,
      when another thread writes data in the wait record being read.
      Reading unsafe data is ok, per design choice, and the code does sanitize
      the data in general, but did not sanitize the length given to memcpy.
      
      The fix is to also sanitize the schema name / object name / file name
      length when extracting the data to produce a row.
[15 Sep 2010 18:23] Marc Alff
See related bug#50557
[15 Sep 2010 19:30] Christopher Powers
Patch approved. I assume that 'm_row' is not subject to concurrent modification as is the 'wait' object.
[15 Sep 2010 19:30] Christopher Powers
Patch approved. I assume that 'm_row' is not subject to concurrent modification as is the 'wait' object.
[15 Sep 2010 21:00] Marc Alff
Pushed into:
- mysql-5.5-bugfixing
- mysql-trunk-bugfixing
- mysql-next-mr-bugfixing
[1 Oct 2010 17:06] Elena Stepanova
Got a very similar stack trace on the recent mysql-next-mr-bugfixing:

  [1] _lwp_kill(0x0, 0xffffffffffffffff, 0xfffffffffffffff1, 0x0, 0xffffffff7f04d574, 0xa), at 0xffffffff7eec8450
=>[2] my_write_core(sig = 11), line 328 in "stacktrace.c"
  [3] handle_segfault(sig = 11), line 2513 in "mysqld.cc"
  [4] __sighndlr(0xb, 0x0, 0xffffffff73f3bd10, 0x1001d58a8, 0x12, 0x0), at 0xffffffff7eec3538
  ---- called from signal handler with signal 11 (SIGSEGV) ------
  [5] memcpy(0x12035e4d8, 0xfffffffedfca1b28, 0xa, 0xfffffffffffffffa, 0x32, 0x11a030b02), at 0xffffffff7effcb28
  [6] table_events_waits_common::make_row(this = 0x12035e430, thread_own_wait = true, pfs_thread = 0x116325778, wait = 0x116458a30), line 275 in "table_events_waits.cc"
  [7] table_events_waits_history::rnd_next(this = 0x12035e430), line 702 in "table_events_waits.cc"
  [8] ha_perfschema::rnd_next(this = 0x11a02fce0, buf = 0x11a0300d0 "^P\xfe\xeeJ"), line 289 in "ha_perfschema.cc"
  [9] handler::ha_rnd_next(this = 0x11a02fce0, buf = 0x11a0300d0 "^P\xfe\xeeJ"), line 2197 in "handler.cc"
  [10] mysql_checksum_table(thd = 0x12258e500, tables = 0x1179a5740, check_opt = 0x122590cc8), line 7308 in "sql_table.cc"
  [11] mysql_execute_command(thd = 0x12258e500), line 2784 in "sql_parse.cc"
  [12] mysql_parse(thd = 0x12258e500, rawbuf = 0x1179a55e0 "CHECKSUM TABLE performance_schema.EVENTS_WAITS_HISTORY EXTENDED", length = 63U, parser_state = 0xffffffff73f3f778), line 5591 in "sql_parse.cc"
  [13] dispatch_command(command = COM_QUERY, thd = 0x12258e500, packet = 0x117bf7be1 "", packet_length = 63U), line 1130 in "sql_parse.cc"
  [14] do_command(thd = 0x12258e500), line 802 in "sql_parse.cc"
  [15] do_handle_one_connection(thd_arg = 0x12258e500), line 1201 in "sql_connect.cc"
  [16] handle_one_connection(arg = 0x12258e500), line 1140 in "sql_connect.cc"
  [17] pfs_spawn_thread(arg = 0x12167c910), line 1078 in "pfs.cc"
[2 Oct 2010 18:12] Bugs System
Pushed into mysql-trunk 5.6.1-m4 (revid:alexander.nozdrin@oracle.com-20101002180948-852x1cuv7c6i85ea) (version source revid:alexander.nozdrin@oracle.com-20101002180857-an32jpuwzemsp4f2) (merge vers: 5.6.1-m4) (pib:21)
[2 Oct 2010 18:13] Bugs System
Pushed into mysql-next-mr (revid:alexander.nozdrin@oracle.com-20101002181053-6iotvl26uurcoryp) (version source revid:alexander.nozdrin@oracle.com-20101002180917-h0n62akupm3z20nt) (pib:21)
[2 Oct 2010 18:16] Bugs System
Pushed into mysql-5.5 5.5.7-rc (revid:alexander.nozdrin@oracle.com-20101002180831-590ka2tuit9qoxbb) (version source revid:alexander.nozdrin@oracle.com-20101002180831-590ka2tuit9qoxbb) (merge vers: 5.5.7-rc) (pib:21)
[4 Nov 2010 1:21] Paul Dubois
Noted in 5.5.7, 5.6.1 changelogs.

The server could crash inside memcpy() when reading certain
Performance Schema tables.