MySQL Bugs: #45243: crash on win in sql thread clear_tables_to

Bug #45243	crash on win in sql thread clear_tables_to_lock() -> free()
Submitted:	1 Jun 2009 12:28	Modified:	16 Sep 2009 9:35
Reporter:	Andrei Elkin	Email Updates:
Status:	Closed	Impact on me:	None
Category:	MySQL Server: Row Based Replication ( RBR )	Severity:	S3 (Non-critical)
Version:	5.1-bugteam	OS:	Windows
Assigned to:	Davi Arnaut	CPU Architecture:	Any
Tags:	rpl_row_basic_3innodb, test failure

Description:
The crash is found on PB2.
In executing
  rpl.rpl_row_basic_3innodb

slave crashed at the end of processing one of two rows events (l.373 of the rpl_row_basic.test)

 --echo [expecting slave to replicate correctly]
  connection master;
  INSERT INTO t4 VALUES (1, "", 1);
  INSERT INTO t4 VALUES (2, repeat(_utf8'a', 128), 2);

Here is the stack:

kernel32!HeapFree
mysqld!free [f:\dd\vctools\crt_bld\self_x86\crt\src\free.c @ 110]
mysqld!Relay_log_info::clear_tables_to_lock [g:\pb2\build\sb_0-555450-1243608735.23\mysql-5.1.36-win-x86\sql\rpl_rli.cc @ 1207]
mysqld!Rows_log_event::do_apply_event [g:\pb2\build\sb_0-555450-1243608735.23\mysql-5.1.36-win-x86\sql\log_event.cc @ 7456]
mysqld!apply_event_and_update_pos [g:\pb2\build\sb_0-555450-1243608735.23\mysql-5.1.36-win-x86\sql\slave.cc @ 2047]
mysqld!exec_relay_log_event [g:\pb2\build\sb_0-555450-1243608735.23\mysql-5.1.36-win-x86\sql\slave.cc @ 2174]
mysqld!handle_slave_sql [g:\pb2\build\sb_0-555450-1243608735.23\mysql-5.1.36-win-x86\sql\slave.cc @ 2891]

How to repeat:
http://pb2.norway.sun.com/web.py?template=mysql_show_test_failure&test_failure_id=2183238

This may relate to bug#40796

A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/78181

3013 Georgi Kodinov	2009-07-08
      Bug#38998, Bug#46029, Bug#45243, Bug#46030 making tests experimental

Pushed into 5.1.37 (revid:joro@sun.com-20090708131116-kyz8iotbum8w9yic) (version source revid:joro@sun.com-20090708073149-qxyfa5vkh1m0pvol) (merge vers: 5.1.37) (pib:11)

Pushed into 5.1.37 (revid:joro@sun.com-20090708131116-kyz8iotbum8w9yic) (version source revid:joro@sun.com-20090708073149-qxyfa5vkh1m0pvol) (merge vers: 5.1.37) (pib:11)

Pushed into 5.4.4-alpha (revid:anozdrin@bk-internal.mysql.com-20090710111017-bnh2cau84ug1hvei) (version source revid:joro@sun.com-20090708121727-rekm6n1iu4vmvcfa) (merge vers: 5.4.4-alpha) (pib:11)

A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

http://lists.mysql.com/commits/80781

2788 Davi Arnaut 2009-08-13
Bug#46013: rpl_extraColmaster_myisam fails on pb2
Bug#45243: crash on win in sql thread clear_tables_to_lock() -> free()
Bug#45242: crash on win in mysql_close() -> free()
Bug#45238: rpl_slave_skip, rpl_change_master failed (lost connection) for STOP SLAVE
Bug#46030: rpl_truncate_3innodb causes server crash on windows
Bug#46014: rpl_stm_reset_slave crashes the server sporadically in pb2

When killing a user session on the server, it's necessary to
interrupt (notify) the thread associated with the session that
the connection is being killed so that the thread is woken up
if waiting for I/O. On a few platforms (Mac, Windows and HP-UX)
where the SIGNAL_WITH_VIO_CLOSE flag is defined, this interruption
procedure is to asynchronously close the underlying socket of
the connection.

In order to enable this schema, each connection serving thread
registers its VIO (I/O interface) so that other threads can
access it and close the connection. But only the owner thread of
the VIO might delete it as to guarantee that other threads won't
see freed memory (the thread unregisters the VIO before deleting
it). A side note: closing the socket introduces a harmless race
that might cause a thread attempt to read from a closed socket,
but this is deemed acceptable.

The problem is that this infrastructure was meant to only be used
by server threads, but the slave I/O thread was registering the
VIO of a mysql handle (a client API structure that represents a
connection to another server instance) as a active connection of
the thread. But under some circumstances such as network failures,
the client API might destroy the VIO associated with a handle at
will, yet the VIO wouldn't be properly unregistered. This could
lead to accesses to freed data if a thread attempted to kill a
slave I/O thread whose connection was already broken.

There was a attempt to work around this by checking whether
the socket was being interrupted, but this hack didn't work as
intended due to the aforementioned race -- attempting to read
from the socket would yield a "bad file descriptor" error.

The solution is to add a hook to the client API that is called
from the client code before the VIO of a handle is deleted.
This hook allows the slave I/O thread to detach the active vio
so it does not point to freed memory.
@ server-tools/instance-manager/mysql_connection.cc
Add stub method required for linking.
@ sql-common/client.c
Invoke hook.
@ sql/client_settings.h
Export hook.
@ sql/slave.cc
Introduce hook that clears the active VIO before it is freed
by the client API.

Queued to 5.0-bugteam

The user visible effect is that a STOP SLAVE statement might lead to a crash on Windows or Mac.

Pushed into 5.1.37-ndb-7.0.8 (revid:jonas@mysql.com-20090826132541-yablppc59e3yb54l) (version source revid:jonas@mysql.com-20090826132541-yablppc59e3yb54l) (merge vers: 5.1.37-ndb-7.0.8) (pib:11)

Pushed into 5.1.37-ndb-6.3.27 (revid:jonas@mysql.com-20090826105955-bkj027t47gfbamnc) (version source revid:jonas@mysql.com-20090826105955-bkj027t47gfbamnc) (merge vers: 5.1.37-ndb-6.3.27) (pib:11)

Pushed into 5.1.37-ndb-6.2.19 (revid:jonas@mysql.com-20090825194404-37rtosk049t9koc4) (version source revid:jonas@mysql.com-20090825194404-37rtosk049t9koc4) (merge vers: 5.1.37-ndb-6.2.19) (pib:11)

Pushed into 5.1.35-ndb-7.1.0 (revid:magnus.blaudd@sun.com-20090827163030-6o3kk6r2oua159hr) (version source revid:jonas@mysql.com-20090826132541-yablppc59e3yb54l) (merge vers: 5.1.37-ndb-7.0.8) (pib:11)

Documented bugfix in the 5.1.37 changelog as follows:

        In some cases, a STOP SLAVE statement could cause the replication slave
        to crash. This issue was specific to MySQL on Windows or Macintosh
        platforms.

Set bug status to Patch Queued waiting for pushes to 5.0/5.4 trees.

Pushed into 5.0.86 (revid:joro@sun.com-20090902102337-n5rw8227wwp5cpx8) (version source revid:davi.arnaut@sun.com-20090813200720-utqy73cj0orcy80z) (merge vers: 5.0.86) (pib:11)

Bugfix also noted in 5.0.86 changelog.

Set status to Patch Pending, waiting for 5.4 push.

Pushed into 5.1.39 (revid:joro@sun.com-20090902154533-8actmfcsjfqovgsb) (version source revid:ramil@mysql.com-20090814091316-07dvnrvaj0th0th2) (merge vers: 5.1.38) (pib:11)

Now documented in the following changelogs: 5.0.86, NDB-6.2.19, NDB-6.3.27, NDB-7.0.8, 5.1.39 (should have documented for Cluster releases, not 5.1.37-main).

Set status to NDI, waiting for push to 5.4.

Pushed into 5.4.4-alpha (revid:alik@sun.com-20090914155317-m1g9wodmndzdj4l1) (version source revid:alik@sun.com-20090914155317-m1g9wodmndzdj4l1) (merge vers: 5.4.4-alpha) (pib:11)

Also documented in the 5.4.4 changelog.

Closed.

Pushed into 5.1.39-ndb-6.3.28 (revid:jonas@mysql.com-20091001055605-ap2kiaarr7p40mmv) (version source revid:jonas@mysql.com-20091001055605-ap2kiaarr7p40mmv) (merge vers: 5.1.39-ndb-6.3.28) (pib:11)

Pushed into 5.1.39-ndb-7.0.9 (revid:jonas@mysql.com-20091001072547-kv17uu06hfjhgjay) (version source revid:jonas@mysql.com-20091001071652-irejtnumzbpsbgk2) (merge vers: 5.1.39-ndb-7.0.9) (pib:11)

Pushed into 5.1.39-ndb-7.1.0 (revid:jonas@mysql.com-20091001123013-g9ob2tsyctpw6zs0) (version source revid:jonas@mysql.com-20091001123013-g9ob2tsyctpw6zs0) (merge vers: 5.1.39-ndb-7.1.0) (pib:11)

Moved 5.4 changelog entry from 5.4.4 to 5.4.3.

Moved 5.4 changelog entry from 5.4.4 to 5.4.3.

Pushed into 5.1.39-ndb-6.2.19 (revid:jonas@mysql.com-20091005103850-dwij2dojwpvf5hi6) (version source revid:jonas@mysql.com-20090930185117-bhud4ek1y0hsj1nv) (merge vers: 5.1.39-ndb-6.2.19) (pib:11)