Bug #120972 Redundant outer DISTINCT creates a second temporary-table deduplication above EXCEPT in MySQL
Submitted: 21 Jul 13:42 Modified: 22 Jul 12:09
Reporter: cl hl Email Updates:
Status: Verified Impact on me:
None 
Category:MySQL Server: Optimizer Severity:S2 (Serious)
Version:8.0.46, 8.4.10, 9.7.1 OS:Any
Assigned to: CPU Architecture:Any

[21 Jul 13:42] cl hl
Description:
### Expected Behaviour

Because `EXCEPT` already produces distinct rows, MySQL should eliminate the outer `DISTINCT`. Both SQL forms should use the same set-operation deduplication without an additional temporary table.

### Actual Behaviour

`EXCEPT` is distinct by default, but MySQL retains the outer DISTINCT as `Temporary table with deduplication` above `Except materialize with deduplication`. Five warm runs had the same 26.1 ms median for both forms, because only 300 rows enter the redundant operator. The extra operator and materialization remain visible and can become costly with wider or higher-cardinality outputs.

How to repeat:
DROP DATABASE IF EXISTS rift_distinct_mre;
CREATE DATABASE rift_distinct_mre;
USE rift_distinct_mre;
CREATE TABLE digits(d INT PRIMARY KEY);
INSERT INTO digits VALUES(0),(1),(2),(3),(4),(5),(6),(7),(8),(9);
CREATE TABLE lhs(id INT PRIMARY KEY,v INT NULL,INDEX(v));
CREATE TABLE rhs(id INT PRIMARY KEY,v INT NULL,INDEX(v));
CREATE TABLE probe(id INT PRIMARY KEY,v INT NULL,INDEX(v));
INSERT INTO lhs SELECT n,IF(n%97=0,NULL,n%1000) FROM
 (SELECT a.d+b.d*10+c.d*100+d.d*1000+e.d*10000+1 n
  FROM digits a,digits b,digits c,digits d,digits e) x;
INSERT INTO rhs SELECT n,IF(n%89=0,NULL,n%700+300) FROM
 (SELECT a.d+b.d*10+c.d*100+d.d*1000+e.d*10000+1 n
  FROM digits a,digits b,digits c,digits d,digits e) x;
INSERT INTO probe SELECT n,IF(n%101=0,NULL,n%1200) FROM
 (SELECT a.d+b.d*10+c.d*100+d.d*1000+e.d*10000+1 n
  FROM digits a,digits b,digits c,digits d,digits e) x;
ANALYZE TABLE lhs,rhs,probe;
EXPLAIN ANALYZE SELECT DISTINCT s.v
FROM ((SELECT v FROM lhs) EXCEPT (SELECT v FROM rhs)) s;
EXPLAIN ANALYZE SELECT s.v
FROM ((SELECT v FROM lhs) EXCEPT (SELECT v FROM rhs)) s;
[22 Jul 12:09] Chaithra Marsur Gopala Reddy
Hi cl hl,

Thank you for the test case. Verified as described. I am marking Bug#120973 as duplicate of this one. When EXCEPT is handled, we will handle INTERSECT as well.

Thanks,
Chaithra