Bug #121026 Semijoin duplicate-weedout missing from plan: EXISTS + NOT EXISTS returns duplicate rows
Submitted: 29 Jul 3:26 Modified: 31 Jul 11:37
Reporter: hiroki nagai Email Updates:
Status: Verified Impact on me:
None 
Category:MySQL Server: Optimizer Severity:S2 (Serious)
Version:8.0.42, 8.0.46, 8.4.9, 9.7.2 OS:Any (official mysql Docker images, linux/arm64 and linux/amd64)
Assigned to: CPU Architecture:Any
Tags: duplicateweedout, semijoin, wrong result

[29 Jul 3:26] hiroki nagai
Description:
## Description

A `SELECT` with a single table in the `FROM` clause, restricted to one row by a
primary-key equality, returns 225 rows instead of 1 when the `WHERE` clause
combines:

1. two nested `EXISTS` subqueries, and
2. a correlated `NOT EXISTS` subquery, and
3. a primary-key constant filter on the outer table,

and the semijoin is executed with the duplicate-weedout strategy.

There is no `GROUP BY`, no `DISTINCT` and no aggregate in the query.

The result cannot be correct under any execution plan. The `FROM` clause names
exactly one table (`a`), and `a.id IN (1)` is a primary-key equality, so the
result is bounded above by one row. `EXISTS` and `NOT EXISTS` are boolean
predicates and have no ability to multiply rows. The 225 returned rows are 225
identical copies of the same row: `COUNT(*) = 225`, `COUNT(DISTINCT id) = 1`.
225 is exactly the number of join combinations produced by flattening the two
`EXISTS` subqueries (5 x 3 x 5 x 3).

`EXPLAIN ANALYZE` shows the `EXISTS` subqueries flattened into plain inner
joins with **no duplicate-removal operator anywhere in the plan**. The antijoin
produced by `NOT EXISTS` behaves correctly and passes one row.

The traditional-format `EXPLAIN` of the same statement, however, **does print
the duplicate-weedout markers**: `Start temporary` / `End temporary` are
present (output in "How to repeat"). So the join optimizer did select the
duplicate-weedout strategy, but the weedout operator is missing from the
executed plan — the strategy is lost between join optimization and iterator
plan construction. Additionally, `Start temporary` is attached to the antijoin
table `e` (the `NOT EXISTS` table), i.e. the weedout range is computed with the
antijoin nest incorrectly included, which may be what prevents the weedout
operator from being installed when the iterator plan is built.

Note that the optimizer's own cost estimate for the root node is `rows=225`,
matching the actual output exactly: the plan as constructed is self-consistently
wrong. The optimizer predicts 225 output rows for a query whose result is
bounded above by one row, i.e. the duplicate-elimination step is already absent
at plan construction time, not skipped during execution.

The `NOT EXISTS` table `e` is **empty**. Its mere presence in the query text is
sufficient to trigger the wrong result; deleting the `NOT EXISTS` clause makes
the query return the correct single row.

While minimizing the test case, each of the following was found to be necessary
to reproduce. Removing any one of them makes the wrong result disappear:

- two nested `EXISTS` subqueries (a single one is not enough)
- the correlated `NOT EXISTS` subquery
- the primary-key constant filter on the outer table (`a.id IN (1)`)
- duplicate-weedout selected as the semijoin strategy

We hit this in production, hint-free, on a query generated by an ORM: a public
API endpoint silently returned 663 identical copies of a single row. The
optimizer chose duplicate-weedout on its own from real table statistics; the
`optimizer_switch` setting in the test case below only pins, on this
deliberately tiny dataset, the strategy that production statistics select
naturally. (On the minimal dataset with default `optimizer_switch`, the cost
model happens to pick FirstMatch and the query returns the correct single row —
the defect is reachable with fully default settings whenever statistics favor
duplicate-weedout, which is how it surfaced in production.)

### Relation to existing reports

This is **not** Bug #120943. That report describes the opposite failure mode in
the same area: the weedout operator is present and over-removes, discarding
valid rows, requires a rowid filesort to feed it, and is reported as fixed in
9.3+. The failure here is that the weedout operator is absent, rows are
duplicated rather than discarded, and it still reproduces on the 9.x line.

Bug #53298 (fixed in 2010, never present in a released 5.6+) is a historical
instance of the same structural failure: "without semijoin strategy, the
semijoin just becomes a plain join, leading to duplicate rows".

### Versions

All tested on official `mysql` Docker images with the exact script below:

| Version | Result |
|---|---|
| 8.0.42 | 225 rows (wrong) |
| 8.0.46 | 225 rows (wrong) |
| 8.4.9 (LTS) | 225 rows (wrong) |
| 9.7.2 | 225 rows (wrong) |

No tested version returns the correct result. The plan shape is identical on
8.0.42 and 9.7.2 (`EXPLAIN ANALYZE` trees below): both are a chain of plain
`Nested loop inner join` nodes with no weedout operator, both estimate and
produce 225 rows. The traditional `EXPLAIN` output is also identical on both,
including the `Start temporary` / `End temporary` markers.

---

How to repeat:
## How to repeat

Run on a fresh server (18 rows of data total):

```sql
DROP DATABASE IF EXISTS weedout_dup;
CREATE DATABASE weedout_dup;
USE weedout_dup;

CREATE TABLE a (id INT PRIMARY KEY);
CREATE TABLE b (id INT PRIMARY KEY, a_id INT, KEY(a_id));
CREATE TABLE c (b_id INT, KEY(b_id));
CREATE TABLE d (b_id INT, KEY(b_id));
CREATE TABLE e (a_id INT, KEY(a_id));

INSERT INTO a VALUES (1),(2),(3);
INSERT INTO b VALUES (1,1),(2,1),(3,1),(4,1),(5,1),
                     (6,2),(7,2),(8,2),(9,2),(10,2),
                     (11,3),(12,3),(13,3),(14,3),(15,3);
INSERT INTO c SELECT id FROM b; INSERT INTO c SELECT id FROM b; INSERT INTO c SELECT id FROM b;
INSERT INTO d SELECT id FROM b; INSERT INTO d SELECT id FROM b; INSERT INTO d SELECT id FROM b;
ANALYZE TABLE a, b, c, d, e;

-- Pin duplicate-weedout. Production statistics select this strategy without any
-- hint or optimizer_switch change.
SET SESSION optimizer_switch='firstmatch=off,loosescan=off,materialization=off';

-- Q1, wrong result. Returns 225 rows; must return 1.
SELECT * FROM a
WHERE EXISTS (SELECT * FROM b WHERE a.id = b.a_id
              AND EXISTS (SELECT * FROM c WHERE c.b_id = b.id))
  AND EXISTS (SELECT * FROM b WHERE a.id = b.a_id
              AND EXISTS (SELECT * FROM d WHERE d.b_id = b.id))
  AND NOT EXISTS (SELECT 1 FROM e WHERE e.a_id = a.id)
  AND a.id IN (1);

-- The 225 rows are 225 copies of the same row (id=1); see the actual row
-- counts in the EXPLAIN ANALYZE output below.
```

(Caveat when measuring: wrapping Q1 in a derived table can change the plan and
mask the bug. Count the rows of Q1 executed as-is, or read the actual row count
off `EXPLAIN ANALYZE`.)

Controls, each returning the correct single row:

- **Q2:** Q1 with `SET SESSION optimizer_switch='duplicateweedout=off'`
- **Q3:** Q1 with `SET SESSION optimizer_switch='semijoin=off'`
- **Q4:** Q1 with the `NOT EXISTS` clause removed
- **Q5:** Q1 with one of the two nested `EXISTS` clauses removed

Traditional-format `EXPLAIN` of Q1 — identical output on 8.0.42 and 9.7.2
(9.x needs `EXPLAIN FORMAT=TRADITIONAL` since the default format changed).
Note `Start temporary` / `End temporary`: the optimizer selected
duplicate-weedout, and the range starts at the antijoin table `e`:

```
+----+-------------+-------+------------+-------+---------------+---------+---------+------------------+------+----------+-------------------------------------------------------+
| id | select_type | table | partitions | type  | possible_keys | key     | key_len | ref              | rows | filtered | Extra                                                 |
+----+-------------+-------+------------+-------+---------------+---------+---------+------------------+------+----------+-------------------------------------------------------+
|  1 | SIMPLE      | a     | NULL       | const | PRIMARY       | PRIMARY | 4       | const            |    1 |   100.00 | Using index                                           |
|  1 | SIMPLE      | e     | NULL       | ref   | a_id          | a_id    | 5       | const            |    1 |   100.00 | Using where; Not exists; Using index; Start temporary |
|  1 | SIMPLE      | b     | NULL       | ref   | PRIMARY,a_id  | a_id    | 5       | const            |    5 |   100.00 | Using index                                           |
|  1 | SIMPLE      | c     | NULL       | ref   | b_id          | b_id    | 5       | repro_check.b.id |    3 |   100.00 | Using index                                           |
|  1 | SIMPLE      | b     | NULL       | ref   | PRIMARY,a_id  | a_id    | 5       | const            |    5 |   100.00 | Using index                                           |
|  1 | SIMPLE      | d     | NULL       | ref   | b_id          | b_id    | 5       | repro_check.b.id |    3 |   100.00 | Using index; End temporary                            |
+----+-------------+-------+------------+-------+---------------+---------+---------+------------------+------+----------+-------------------------------------------------------+
```

`EXPLAIN ANALYZE` (FORMAT=TREE with actual row counts) of Q1 on **8.0.42**.
Note the absence of any "Remove duplicate ... using temporary table (weedout)"
operator, and the root producing 225 actual rows:

```
-> Nested loop inner join  (cost=35.4 rows=225) (actual time=0.0168..0.0988 rows=225 loops=1)
    -> Nested loop inner join  (cost=11.6 rows=75) (actual time=0.0113..0.0293 rows=75 loops=1)
        -> Nested loop inner join  (cost=3.85 rows=15) (actual time=0.01..0.0148 rows=15 loops=1)
            -> Nested loop inner join  (cost=1.1 rows=5) (actual time=0.00742..0.00804 rows=5 loops=1)
                -> Nested loop antijoin  (cost=0.35 rows=1) (actual time=0.00504..0.00517 rows=1 loops=1)
                    -> Rows fetched before execution  (cost=0..0 rows=1) (actual time=41e-6..41e-6 rows=1 loops=1)
                    -> Limit: 1 row(s)  (cost=0.35 rows=1) (actual time=0.00433..0.00433 rows=0 loops=1)
                        -> Covering index lookup on e using a_id (a_id=1)  (cost=0.35 rows=1) (actual time=0.00379..0.00379 rows=0 loops=1)
                -> Covering index lookup on b using a_id (a_id=1)  (cost=0.751 rows=5) (actual time=0.00225..0.00254 rows=5 loops=1)
            -> Covering index lookup on c using b_id (b_id=b.id)  (cost=4.66 rows=3) (actual time=775e-6..0.00124 rows=3 loops=5)
        -> Covering index lookup on b using a_id (a_id=1)  (cost=0.751 rows=5) (actual time=564e-6..825e-6 rows=5 loops=15)
    -> Covering index lookup on d using b_id (b_id=b.id)  (cost=4.66 rows=3) (actual time=383e-6..786e-6 rows=3 loops=75)
```

Same statement on **9.7.2** — identical plan shape, identical wrong result:

```
-> Nested loop inner join  (cost=35.4 rows=225) (actual time=0.0512..0.115 rows=225 loops=1)
    -> Nested loop inner join  (cost=11.6 rows=75) (actual time=0.0495..0.0639 rows=75 loops=1)
        -> Nested loop inner join  (cost=3.85 rows=15) (actual time=0.0485..0.0525 rows=15 loops=1)
            -> Nested loop inner join  (cost=1.1 rows=5) (actual time=0.0466..0.0472 rows=5 loops=1)
                -> Nested loop antijoin  (cost=0.35 rows=1) (actual time=0.0443..0.0444 rows=1 loops=1)
                    -> Rows fetched before execution  (cost=0..0 rows=1) (actual time=42e-6..84e-6 rows=1 loops=1)
                    -> Limit: 1 row(s)  (cost=0.35 rows=1) (actual time=0.00354..0.00354 rows=0 loops=1)
                        -> Covering index lookup on e using a_id (a_id = 1)  (cost=0.35 rows=1) (actual time=0.00287..0.00287 rows=0 loops=1)
                -> Covering index lookup on b using a_id (a_id = 1)  (cost=0.751 rows=5) (actual time=0.00217..0.00237 rows=5 loops=1)
            -> Covering index lookup on c using b_id (b_id = b.id)  (cost=4.66 rows=3) (actual time=541e-6..908e-6 rows=3 loops=5)
        -> Covering index lookup on b using a_id (a_id = 1)  (cost=0.751 rows=5) (actual time=369e-6..603e-6 rows=5 loops=15)
    -> Covering index lookup on d using b_id (b_id = b.id)  (cost=4.66 rows=3) (actual time=223e-6..537e-6 rows=3 loops=75)
```

For contrast, the default-`optimizer_switch` plan on 9.7.2 for the same query —
here the semijoin nests carry a FirstMatch strategy and the result is the
correct single row:

```
-> Nested loop semijoin (FirstMatch)  (cost=24.9 rows=225) (actual time=0.12..0.12 rows=1 loops=1)
    -> Nested loop semijoin (FirstMatch)  (cost=2.1 rows=15) (actual time=0.115..0.115 rows=1 loops=1)
        -> Nested loop antijoin  (cost=0.35 rows=1) (actual time=0.0753..0.0753 rows=1 loops=1)
            -> Rows fetched before execution  (cost=0..0 rows=1) (actual time=0.0293..0.0293 rows=1 loops=1)
            -> Covering index lookup on e using a_id (a_id = 1)  (cost=0.35 rows=1) (actual time=0.00392..0.00392 rows=0 loops=1)
        -> Nested loop inner join  (cost=3.5 rows=15) (actual time=0.00567..0.00567 rows=1 loops=1)
            -> Covering index lookup on b using a_id (a_id = 1)  (cost=0.751 rows=5) (actual time=0.00288..0.00288 rows=1 loops=1)
            -> Covering index lookup on c using b_id (b_id = b.id)  (cost=4.66 rows=3) (actual time=0.00267..0.00267 rows=1 loops=1)
    -> Nested loop inner join  (cost=3.5 rows=15) (actual time=0.00371..0.00371 rows=1 loops=1)
        -> Covering index lookup on b using a_id (a_id = 1)  (cost=0.751 rows=5) (actual time=0.00137..0.00137 rows=1 loops=1)
        -> Covering index lookup on d using b_id (b_id = b.id)  (cost=4.66 rows=3) (actual time=0.00192..0.00192 rows=1 loops=1)
```

Suggested fix:
Ensure the duplicate-weedout strategy selected by the join optimizer (visible
as `Start temporary` / `End temporary` in traditional `EXPLAIN`) is actually
installed in the iterator plan, and that the weedout range is computed without
including the antijoin nest. Bug #53298 is a historical instance of the same
class, where
stale bookkeeping of duplicate-producing tables during plan search led to a
chosen plan carrying no semijoin strategy at all.

Workaround: `SET SESSION optimizer_switch='duplicateweedout=off'`, at the cost
of losing the strategy for every other query on the server.
[29 Jul 3:57] hiroki nagai
[Part 1/2: correction to the necessary conditions + smaller test case]

Correction to the "necessary conditions" list in the original report, plus a
smaller test case. A root-cause analysis from reading the 8.0.42 source follows
in the next comment.

(Note on the database name: the script in the original report creates
`weedout_dup`, but the `EXPLAIN` outputs pasted there were taken in a database
named `repro_check`, which is why the `ref` column shows `repro_check.b.id`.
Everything below consistently uses `repro_check`.)

1. The list of necessary conditions in the original report is wrong
-------------------------------------------------------------------

I listed four conditions as each being necessary. Two of them are not. They were
confounds: removing either one happened to make the cost model pick FirstMatch
instead of duplicate-weedout on that particular dataset, so the wrong result
disappeared for a reason unrelated to the condition itself.

With the strategy pinned to duplicate-weedout in every case, measured on 8.0.42:

  Q1 from the original report                          -> 225 rows (correct: 1)
  Q1 with a.id IN (1) replaced by a range predicate    -> 225 rows (correct: 1)
  Q1 with the filter on a removed entirely             -> 675 rows (correct: 3)
  Q1 with one nested EXISTS removed                    ->  15 rows (correct: 1)
  Flat single EXISTS, three tables only                ->   5 rows (correct: 1)

So neither the primary-key constant filter nor the second nested EXISTS is
required. The revised list of necessary conditions is:

- one EXISTS subquery (nesting is not required)
- one correlated NOT EXISTS subquery
- a duplicate-weedout range whose first table is the antijoin table

Note the third row: with no filter on `a` at all, the result is 675 = 3 x 225,
i.e. exactly (number of outer rows) x (fan-out). No deduplication of any kind is
taking place. This matters for the root-cause analysis in the next comment.

2. Smaller test case: three tables, one EXISTS, one NOT EXISTS
--------------------------------------------------------------

DROP DATABASE IF EXISTS repro_check;
CREATE DATABASE repro_check;
USE repro_check;

CREATE TABLE a (id INT PRIMARY KEY);
CREATE TABLE b (id INT PRIMARY KEY, a_id INT, KEY(a_id));
CREATE TABLE e (a_id INT, KEY(a_id));

INSERT INTO a VALUES (1),(2),(3);
INSERT INTO b VALUES (1,1),(2,1),(3,1),(4,1),(5,1),
                     (6,2),(7,2),(8,2),(9,2),(10,2),
                     (11,3),(12,3),(13,3),(14,3),(15,3);
-- e is left empty
ANALYZE TABLE a, b, e;

SET SESSION optimizer_switch='firstmatch=off,loosescan=off,materialization=off';

-- Returns 5 rows. Must return 1.
SELECT * FROM a
WHERE EXISTS (SELECT * FROM b WHERE a.id = b.a_id)
  AND NOT EXISTS (SELECT 1 FROM e WHERE e.a_id = a.id)
  AND a.id IN (1);

Five identical copies of the single row (1). Table `e` is empty; deleting the
NOT EXISTS line makes the query return the correct single row. Traditional
EXPLAIN shows `Start temporary` on `e` and `End temporary` on `b`; EXPLAIN
ANALYZE shows no weedout operator and a `Limit: 1 row(s)` above the lookup on
`e`, inside the antijoin.

This is a much more ordinary query shape than the one in the original report:
a single-table select with one EXISTS permission-style predicate and one
NOT EXISTS exclusion predicate. That combination is what ORMs generate
routinely, which suggests the exposure is considerably wider than the original
five-table case implied.
[29 Jul 3:57] hiroki nagai
[Part 2/2: root cause in the 8.0.42 source + suggested fix]

The following is from reading the 8.0.42 source. I have not confirmed it under a
debugger, but it accounts for every measurement in the previous comment,
including the 675 case.

Step 1: the weedout range starts at the antijoin table `e` (`Start temporary`),
and the outer table `a` sits in the prefix, before the range. I originally
attributed this to `a` being a const table. That was wrong: the wrong result
persists when `a` is read through a range scan. `a` is outside the weedout
range because of its position in the plan, not its access type, so it is never
a candidate for the deduplication key either way.

Step 2: setup_semijoin_dups_elimination() (sql/sql_select.cc) walks the range
and collects the tables whose row IDs form the deduplication key, via
sj_table_is_included(), which starts with:

  static bool sj_table_is_included(JOIN *join, JOIN_TAB *join_tab) {
    if (join_tab->emb_sj_nest) return false;

`b` is excluded because it is semijoin-inner. `e` is excluded too, because
antijoin nests are marked with the same flag as semijoin nests: sql/table.h
distinguishes them only via is_sj_nest()/is_aj_nest() on top of the shared
m_is_sj_or_aj_nest, and JOIN::set_semijoin_embedding() sets emb_sj_nest to
"the closest semi-join or anti-join nest". (For the same reason, the
assert(tab_in_sj_nest) at the top of the SJ_OPT_DUPS_WEEDOUT case passes on
debug builds, so the unusual shape of this range - first table not
semijoin-inner - is not caught there either.) With `a` outside the range and
both in-range tables excluded, the key table list is empty.

Step 3: create_sj_tmp_table() (sql/sql_select.cc) infers confluence from the
accumulated row ID width:

  if (jt_rowid_offset) {          /* Temptable has at least one rowid */
    sjtbl->is_confluent = false;
    ...
  } else {
    /*
      This is confluent case where the entire subquery predicate does
      not depend on anything at all, ie this is
        WHERE const IN (uncorrelated select)
    */
    sjtbl->is_confluent = true;

The comment states what the branch was written for: an uncorrelated predicate.
That is not the situation here - the predicates are correlated and the semijoin
does fan out; the key is empty only because no table in the range was eligible
to contribute a row ID. The two cases are indistinguishable at this point, and
the second is silently treated as the first.

Step 4: this is where the deduplication is actually lost. In FindSubstructure()
(sql/sql_executor.cc), the helper ConnectJoins() uses to classify each table
range before building iterators:

  if (is_outer_join && is_weedout) {
    if (outer_join_end > weedout_end) {
      is_weedout = false;                 // handled at a lower level
    } else {
      if (qep_tab->flush_weedout_table->is_confluent) {
        // We have the case where the right side of an outer join is a
        // confluent weedout. The weedout will return at most one row, so
        // replace the weedout with LIMIT 1.
        *add_limit_1 = true;
      } else {
        MarkUnhandledDuplicates(qep_tab->flush_weedout_table, this_idx,
                                weedout_end, unhandled_duplicates);
      }
      is_weedout = false;
    }
  }

is_outer_join is true here because `e` is the first inner table of the outer
join used to execute the antijoin. When the non-confluent path (or the
analogous is_semijoin branch below) drops a weedout, it calls
MarkUnhandledDuplicates() first, so the top level installs a compensating
weedout. The confluent shortcut does not: it sets *add_limit_1 and records
nothing. Combined with step 3:

- the weedout is dropped,
- no compensating weedout is added at the top level,
- ConnectJoins() adds the LIMIT 1 to the inner side of the antijoin -
  the "Limit: 1 row(s)" above the lookup on `e` in the broken plans.

The comment's premise, "The weedout will return at most one row", is false
here, because is_confluent was set for the wrong reason in step 3.

Why the 675 case confirms this: if the substituted LIMIT 1 wrapped the weedout
range subtree, the no-filter query would return one row per outer row, i.e. 3.
If it wrapped the whole join output, it would return 1. The measured value is
675 = 3 x 225, the full undeduplicated product. So the LIMIT 1 lands somewhere
that constrains nothing (the inner side of an antijoin against an empty table),
and no deduplication happens at any level - consistent with step 4, and not
with the weedout merely being misplaced.

Suggested fix
-------------

(a) The root fix, in create_sj_tmp_table(): stop inferring confluence from
jt_rowid_offset == 0. Pass the genuinely-confluent case
(WHERE const IN (uncorrelated select)) in explicitly from the caller, which
knows it. An empty deduplication key arising from an ineligible range is a
different condition and should leave is_confluent false. With is_confluent
false, step 4 takes the existing else branch, MarkUnhandledDuplicates() runs,
and the top level installs a weedout keyed on the tables outside the range -
`a` - which is the correct key. No new mechanism is needed.

(b) A narrower, defensive change, in FindSubstructure(): have the is_confluent
shortcut also call MarkUnhandledDuplicates(), making the is_outer_join branch
symmetric with the is_semijoin branch below it.

I would expect (a) to be the correct fix. (b) alone would restore correct
results in this case - the compensating weedout keyed on `a` performs the
missing deduplication - but it over-corrects the genuinely confluent case,
where the LIMIT 1 substitution is already correct and (b) would additionally
install a redundant top-level weedout. A third possibility, extending the
weedout range to begin before the antijoin table so it includes a key-eligible
table, would also avoid the empty-key condition, but it changes plan shapes
more broadly and I have not evaluated the consequences.

This is a description of the defect rather than a patch; I have not signed the
OCA. If a patch would be useful I am willing to look into arranging that, but I
would want to confirm the intended direction first, since the choice between
(a) and the range-extension option is not one I can make from outside.
[31 Jul 11:37] Chaithra Marsur Gopala Reddy
Hi hiroki nagai,

Thank you for the test case. Verified as described.