Bug #5987 subselect in bool function crashes server (prepared statements)

• Submitted: 8 Oct 2004 16:33
• Modified: 9 Oct 2004 22:44
• Reporter: Konstantin Osipov (OCA)
• Status: Closed
• Category: MySQL Server
• Severity: S3 (Non-critical)
• Version: 4.1.*
• OS: Any (All)
• Assigned to: Konstantin Osipov
• CPU Architecture: Any

[8 Oct 2004 16:33] Konstantin Osipov

Description:
The following query in prepared mode:

create table t1 (a int);
prepare stmt from "select * from t1 where 1 > (1 in (SELECT * FROM t1))";
execute stmt;

makes the server to crash.

How to repeat:
See description

Suggested fix:
Rewrite Item_bool_rowready_func2 to not restore its arguments if they were changed 
to point to a transformed subquery.

[8 Oct 2004 22:53] Konstantin Osipov

mail:/tmp/step3.context.diff

[9 Oct 2004 22:44] Konstantin Osipov

Fixed in 4.1.6:
  bk commit - 4.1 tree (konstantin:1.2069) BUG#5987

ChangeSet
  1.2069 04/10/10 02:39:22 [e-mail] +7 -0
  A fix and test case for Bug#5987 "subselect in bool function
  crashes server (prepared statements)": the bug was that all boolean
  items always recovered its original arguments at statement cleanup
  stage.
  This collided with Item_subselect::select_transformer, which tries to
  permanently change the item tree to use a transformed subselect instead of
  original one.
  So we had this call sequence for prepare:
  mysql_stmt_prepare -> JOIN::prepare ->
  Item_subselect::fix_fields -> the item tree gets transformed ->
  Item_bool_rowready_func2::cleanup, item tree is recovered to original
  state, while it shouldn't have been;
  mysql_stmt_execute -> attempts to execute a broken tree -> crash.
  Now instead of bluntly recovering all arguments of bool functions in
  Item_bool_rowready_func2::cleanup, we recover only those
  which were changed, and do it in one place.
  There still would exist a possibility for a collision with subselect
  transformation, if permanent and temporary changes were performed at the
  same stage.
  But fortunately subselect transformation is always done first, so it
  doesn't conflict with the optimization done by propogate_cond_constants.
  Now we have:
  mysql_stmt_prepare -> JOIN::prepare -> subselect transformation
  permanently changes the tree -> cleanup doesn't recover anything,
  because nothing was registered for recovery.
  mysql_stmt_execute -> JOIN::prepare (the tree is already transformed,
  so it doesn't change), JOIN::optimize ->
  propogate_cond_constants -> temporarily changes the item tree
  with constants -> JOIN::execute -> cleanup ->
  the changes done by propogate_cond_constants are recovered, as
 they were registered for recovery.