Bug #121000 Incorrect index lookup range pushdown for (decimal constant NOT IN (indexed_int_column)) negation causes duplicate rows
Submitted: 23 Jul 15:43 Modified: 28 Jul 13:55
Reporter: Xiaoyuan Xie Email Updates:
Status: Duplicate Impact on me:
None 
Category:MySQL Server: Optimizer Severity:S2 (Serious)
Version:9.6.0,9.7.0 OS:Ubuntu
Assigned to: CPU Architecture:Any

[23 Jul 15:43] Xiaoyuan Xie
Description:
A query result divergence occurs during TLP (Ternary Logic Partitioning) evaluation when a predicate compares a decimal value (produced by a CASE expression) with a NOT IN operator against an indexed TINYINT ZEROFILL column.Specifically:For a table containing a single row c = 0 (TINYINT ZEROFILL), CASE NULL WHEN 1 THEN 0 ELSE 0.1 END evaluates to 0.1.Evaluating 0.1 NOT IN (c) yields TRUE since $0.1 \neq 0$. Thus, WHERE P returns 0.However, when evaluating the negated condition WHERE NOT (0.1 NOT IN (c)) (equivalent to 0.1 IN (c)), MySQL's optimizer improperly truncates 0.1 to 0 during index range/lookup construction, mistakenly concluding 0.1 IN (0) is TRUE.As a result, the value 0 satisfies both WHERE P and WHERE NOT P, returning 2 rows for a 1-row table under TLP partitioning.

How to repeat:
DROP TABLE IF EXISTS t;
CREATE TABLE t(c TINYINT ZEROFILL);
CREATE INDEX i ON t(c);

INSERT INTO t VALUES (0);

-- Baseline Query (Returns 1 row: 000)
SELECT c FROM t;

-- TLP Partitioning Query (Incorrectly returns 2 rows)
SELECT c FROM t
WHERE
    (CASE NULL WHEN 1 THEN 0 ELSE 0.1 END) NOT IN (c)

UNION ALL

SELECT c FROM t
WHERE NOT (
    (CASE NULL WHEN 1 THEN 0 ELSE 0.1 END) NOT IN (c)
)

UNION ALL

SELECT c FROM t
WHERE (
    (CASE NULL WHEN 1 THEN 0 ELSE 0.1 END) NOT IN (c)
) IS UNKNOWN;

Expected Result:

The TLP partitioning query should return exactly 1 row (0), matching the baseline table count.

Actual Result:

The TLP query returns 2 rows (0 and 0), demonstrating that row 0 incorrectly satisfies both the positive and negated predicates simultaneously due to index range coercion.
[28 Jul 13:55] Chaithra Marsur Gopala Reddy
Hi Xiaoyuan Xie,

Thank you for the test case. Verified as described. This is fixed in the upcoming 9.7 LTS release as part of the fix for Bug#120786.

Thanks,
Chaithra