Bug #120226 Hypergraph optimizer: ROR intersect OverflowBitset capacity mismatch causes server crash (SIGABRT/SIGSEGV)
Submitted: 7 Apr 14:59 Modified: 8 Apr 4:23
Reporter: Yakir Gibraltar (OCA) Email Updates:
Status: Verified Impact on me:
None 
Category:MySQL Server: Optimizer Severity:S2 (Serious)
Version:9.7.0-er2 OS:Linux (Ubuntu 22.04 (Docker, native x86_64))
Assigned to: CPU Architecture:x86

[7 Apr 14:59] Yakir Gibraltar
Description:
When the hypergraph optimizer evaluates a ROR (Row-ID Ordered) index merge intersection plan on a query with enough sargable join predicates to push the total predicate count past 63, the server crashes during query optimization — no rows are ever read.

- Debug build: SIGABRT (assertion failure in OverflowBitset::Or at overflow_bitset.h:285)
- Release build: OOM error (~1 GB allocation attempt) followed by SIGSEGV

The crash is in ProposeRowIdOrderedIntersect() → UpdateAbsorbedPredicates() → OverflowBitset::Or(), triggered by a bitset capacity mismatch between the absorbed_predicates bitset and the scan's predicates bitset.

WORKAROUND
----------
  SET optimizer_switch='index_merge_intersection=off';
  -- or --
  SET optimizer_switch='hypergraph_optimizer=off';

How to repeat:
Tested on: MySQL 9.7.0-er2-debug (cmake -DCMAKE_BUILD_TYPE=Debug), native x86_64 Linux (Ubuntu 22.04, Docker).
Debug build: SIGABRT. Release build: OOM ([ERROR] Out of memory (Needed ~1069199880 bytes)) then SIGSEGV.

-- SETUP --

CREATE DATABASE hg_ror;
USE hg_ror;

-- Main table: 25 single-column indexes (a,b used in WHERE; c-y used for sargable join preds)
CREATE TABLE big_indexed (
  id INT PRIMARY KEY AUTO_INCREMENT,
  a INT, b INT, c INT, d INT, e INT,
  f INT, g INT, h INT, i INT, j INT,
  k INT, l INT, m INT, n INT, o INT,
  p INT, q INT, r INT, s INT, t INT,
  u INT, v INT, w INT, x INT, y INT,
  KEY(a), KEY(b), KEY(c), KEY(d), KEY(e),
  KEY(f), KEY(g), KEY(h), KEY(i), KEY(j),
  KEY(k), KEY(l), KEY(m), KEY(n), KEY(o),
  KEY(p), KEY(q), KEY(r), KEY(s), KEY(t),
  KEY(u), KEY(v), KEY(w), KEY(x), KEY(y)
);

INSERT INTO big_indexed (a,b,c,d,e,f,g,h,i,j,k,l,m,n,o,p,q,r,s,t,u,v,w,x,y)
WITH RECURSIVE s(n) AS (SELECT 1 UNION ALL SELECT n+1 FROM s WHERE n<1000)
SELECT n%10,n%10,n%10,n%10,n%10,n%10,n%10,n%10,n%10,n%10,
       n%10,n%10,n%10,n%10,n%10,n%10,n%10,n%10,n%10,n%10,
       n%10,n%10,n%10,n%10,n%10 FROM s;

ANALYZE TABLE big_indexed;

-- Reference table (no indexes needed): column names match big_indexed cols c-y (excluding t)
CREATE TABLE ref_y (
  c INT, d INT, e INT, f INT, g INT, h INT, i INT, j INT, k INT, l INT,
  m INT, n INT, o INT, p INT, q INT, r INT, s INT, u INT, v INT,
  w INT, x INT, y INT
);
INSERT INTO ref_y VALUES (1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1);

-- CRASH TRIGGER --
-- Predicate accounting:
--   num_filter_predicates = 2  (t.a=1, t.b=1)           -> inline bitset (2 <= 63)
--   sargable join preds   = 66 (22 cols * 3 aliases)     -> predicates.size() = 68 > 63
--   -> overflow bitset for scan.predicates
--   -> assert(a.is_inline() == b.is_inline()) fires in UpdateAbsorbedPredicates()

SET optimizer_switch='hypergraph_optimizer=on,index_merge=on,index_merge_intersection=on';

SELECT 1
FROM big_indexed t
JOIN ref_y j1 USING(c,d,e,f,g,h,i,j,k,l,m,n,o,p,q,r,s,u,v,w,x,y)
JOIN ref_y j2 USING(c,d,e,f,g,h,i,j,k,l,m,n,o,p,q,r,s,u,v,w,x,y)
JOIN ref_y j3 USING(c,d,e,f,g,h,i,j,k,l,m,n,o,p,q,r,s,u,v,w,x,y)
WHERE t.a=1 AND t.b=1
LIMIT 1;

-- RESULT: Server crashes immediately during optimization. No rows are read.

mysqld: /src/sql/join_optimizer/overflow_bitset.h:285: static MutableOverflowBitset OverflowBitset::Or(MEM_ROOT*, OverflowBitset, OverflowBitset): Assertion `a.capacity() == b.capacity()' failed.
2026-04-07T14:49:00Z UTC - mysqld got signal 6 ;
Most likely, you have hit a bug, but this error can also be caused by malfunctioning hardware.
BuildID[sha1]=c0657c6f503c026323cc595c51aaadbca1d27258
Thread pointer: 0x7f20dc00d0f0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 7f21d46f6998 thread_stack 0x100000
 #0 0x564b19257b9f print_fatal_signal at /src/sql/signal_handler.cc:321
 #1 0x564b19257e2e _Z19handle_fatal_signaliP9siginfo_tPv at /src/sql/signal_handler.cc:403
 #2 0x7f21ecf7251f <unknown>
 #3 0x7f21ecfc69fc <unknown>
 #4 0x7f21ecf72475 <unknown>
 #5 0x7f21ecf587f2 <unknown>
 #6 0x7f21ecf5871a <unknown>
 #7 0x7f21ecf69e95 <unknown>
 #8 0x564b18ddf739 _ZN14OverflowBitset2OrEP8MEM_ROOTS_S_ at /src/sql/join_optimizer/overflow_bitset.h:285
 #9 0x564b1971335b UpdateAbsorbedPredicates at /src/sql/join_optimizer/join_optimizer.cc:2988
 #10 0x564b19713c12 ProposeRowIdOrderedIntersect at /src/sql/join_optimizer/join_optimizer.cc:3131
 #11 0x564b19713892 ProposeAllRowIdOrderedIntersectPlans at /src/sql/join_optimizer/join_optimizer.cc:3080
 #12 0x564b197127a2 ProposeAllIndexMergeScans at /src/sql/join_optimizer/join_optimizer.cc:2846
 #13 0x564b19710b22 FindIndexRangeScans at /src/sql/join_optimizer/join_optimizer.cc:2516
 #14 0x564b1970d80f FindRangeScans at /src/sql/join_optimizer/join_optimizer.cc:1648
 #15 0x564b1970e3cb FoundSingleNode at /src/sql/join_optimizer/join_optimizer.cc:1847
 #16 0x564b19730869 EnumerateAllConnectedPartitions<(anonymous namespace)::CostingReceiver> at /src/sql/join_optimizer/subgraph_enumeration.h:669
 #17 0x564b1972c4ba FindBestQueryPlanInner at /src/sql/join_optimizer/join_optimizer.cc:9425
 #18 0x564b1972e509 _Z17FindBestQueryPlanP3THDP11Query_block at /src/sql/join_optimizer/join_optimizer.cc:9904
 #19 0x564b18f7500b _ZN4JOIN8optimizeEb at /src/sql/sql_optimizer.cc:652
 #20 0x564b1905745b _ZN11Query_block8optimizeEP3THDb at /src/sql/sql_select.cc:2141
 #21 0x564b1911ead2 _ZN16Query_expression8optimizeEP3THDP5TABLEb at /src/sql/sql_union.cc:523
 #22 0x564b19054c41 _ZN11Sql_cmd_dml13execute_innerEP3THD at /src/sql/sql_select.cc:1126
 #23 0x564b19053b2d _ZN11Sql_cmd_dml7executeEP3THD at /src/sql/sql_select.cc:798
 #24 0x564b18fba496 _Z21mysql_execute_commandP3THDb at /src/sql/sql_parse.cc:4766
 #25 0x564b18fbca9e _Z20dispatch_sql_commandP3THDP12Parser_stateb at /src/sql/sql_parse.cc:5441
 #26 0x564b18fb21e4 _Z16dispatch_commandP3THDPK8COM_DATA19enum_server_command at /src/sql/sql_parse.cc:2152
 #27 0x564b18fb000b _Z10do_commandP3THD at /src/sql/sql_parse.cc:1491
 #28 0x564b1923e0ea handle_connection at /src/sql/conn_handler/connection_handler_per_thread.cc:304
 #29 0x564b1b330d0e pfs_spawn_thread at /src/storage/perfschema/pfs.cc:3067
 #30 0x7f21ecfc4ac2 <unknown>
 #31 0x7f21ed055a83 <unknown>
 #32 0xffffffffffffffff <unknown>

Trying to get some variables.
Some pointers may be invalid and cause the dump to abort.
Query (7f20dc158d90): SELECT 1 FROM big_indexed t JOIN ref_y j1 USING(c,d,e,f,g,h,i,j,k,l,m,n,o,p,q,r,s,u,v,w,x,y) JOIN ref_y j2 USING(c,d,e,f,g,h,i,j,k,l,m,n,o,p,q,r,s,u,v,w,x,y) JOIN ref_y j3 USING(c,d,e,f,g,h,i,j,k,l,m,n,o,p,q,r,s,u,v,w,x,y) WHERE t.a=1 AND t.b=1 LIMIT 1
Connection ID (thread ID): 18
Status: NOT_KILLED

The manual page at https://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you find out what is causing the crash.

Suggested fix:
Thank you, Yakir Gibraltar
[8 Apr 4:23] Chaithra Marsur Gopala Reddy
Hi Yakir Gibraltar,

Thank you for the test case. Verified as described.