Bug #120973 Redundant outer DISTINCT creates a second temporary-table deduplication above INTERSECT in MySQL
Submitted: 21 Jul 13:44 Modified: 22 Jul 12:10
Reporter: cl hl Email Updates:
Status: Duplicate Impact on me:
None 
Category:MySQL Server: Optimizer Severity:S2 (Serious)
Version:9.7.1 OS:Any
Assigned to: CPU Architecture:Any

[21 Jul 13:44] cl hl
Description:
### Expected Behaviour

Because `INTERSECT` already produces a duplicate-free result, MySQL should remove the outer `DISTINCT`. The original and simplified queries should have the same execution plan, with only the deduplication required by `INTERSECT`.

### Actual Behaviour

`INTERSECT` already returns distinct rows, so the outer `DISTINCT` is redundant. MySQL nevertheless adds `Temporary table with deduplication` above `Intersect materialize with deduplication`.

Five warm `EXPLAIN ANALYZE` runs gave median root times of 24.9 ms with the redundant DISTINCT and 25.7 ms without it (-3.1%). This small reversed difference is runtime noise because only 701 rows reach the extra operator. The plan-level inefficiency is deterministic; larger set-operation outputs can make its materialization cost visible.

How to repeat:
DROP DATABASE IF EXISTS rift_distinct_mre;
CREATE DATABASE rift_distinct_mre;
USE rift_distinct_mre;
CREATE TABLE digits(d INT PRIMARY KEY);
INSERT INTO digits VALUES(0),(1),(2),(3),(4),(5),(6),(7),(8),(9);
CREATE TABLE lhs(id INT PRIMARY KEY,v INT NULL,INDEX(v));
CREATE TABLE rhs(id INT PRIMARY KEY,v INT NULL,INDEX(v));
CREATE TABLE probe(id INT PRIMARY KEY,v INT NULL,INDEX(v));
INSERT INTO lhs SELECT n,IF(n%97=0,NULL,n%1000) FROM
 (SELECT a.d+b.d*10+c.d*100+d.d*1000+e.d*10000+1 n
  FROM digits a,digits b,digits c,digits d,digits e) x;
INSERT INTO rhs SELECT n,IF(n%89=0,NULL,n%700+300) FROM
 (SELECT a.d+b.d*10+c.d*100+d.d*1000+e.d*10000+1 n
  FROM digits a,digits b,digits c,digits d,digits e) x;
INSERT INTO probe SELECT n,IF(n%101=0,NULL,n%1200) FROM
 (SELECT a.d+b.d*10+c.d*100+d.d*1000+e.d*10000+1 n
  FROM digits a,digits b,digits c,digits d,digits e) x;
ANALYZE TABLE lhs,rhs,probe;
EXPLAIN ANALYZE SELECT DISTINCT s.v
FROM ((SELECT v FROM lhs) INTERSECT (SELECT v FROM rhs)) s;
EXPLAIN ANALYZE SELECT s.v
FROM ((SELECT v FROM lhs) INTERSECT (SELECT v FROM rhs)) s;
[22 Jul 12:10] Chaithra Marsur Gopala Reddy
Hi cl hl,

Thank you for the test case. Verified as described. However marking this as duplicate of Bug#120972. When the problem is addressed, both EXCEPT and INTERSECT would be handled together.

Thanks,
Chaithra