Bug #121683 With strict vm.overcommit_memory=2 kernel setting, MySQL crash with 'std::bad_alloc' on Rapid_json path

Description:
With a strict overcommit kernel setting, on a system where the committed (don't confuse with actually used) memory gets close to the commit limit, new memory allocations from MySQL are rejected by the OS, resulting in the usual errors, like:
2026-11-05T16:59:23.290284Z 0 [ERROR] [MY-010249] [Server] Can't create thread to handle new connection(errno= 11)
or seen in the dmesg log:
[Thu Nov  5 16:59:19 2026] __vm_enough_memory: pid: 37087, comm: mysqld, not enough memory for the allocation
[Thu Nov  5 17:01:23 2026] __vm_enough_memory: pid: 37167, comm: connection, not enough memory for the allocation

However, an example JSON object insert triggers a crash instead of gracefully handling the malloc refusal:

terminate called after throwing an instance of 'std::bad_alloc'
  what():  std::bad_alloc
2026-11-05T17:04:08Z UTC - mysqld got signal 6 ;
Most likely, you have hit a bug, but this error can also be caused by malfunctioning hardware.
BuildID[sha1]=381e3397545ef3903831e5caff813f57fd217d99
Thread pointer: 0x7fde18ece1b0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 7fde306f5bc0 thread_stack 0x100000
 #0 0x1832b70 _Z19handle_fatal_signaliP9siginfo_tPv at mysql-8.4.13/sql/signal_handler.cc:401
 #1 0x1832b70 _Z19handle_fatal_signaliP9siginfo_tPv at mysql-8.4.13/sql/signal_handler.cc:391
 #2 0x7fde4803e72f <unknown>
 #3 0x7fde4808ba6c <unknown>
 #4 0x7fde4803e685 <unknown>
 #5 0x7fde48028832 <unknown>
 #6 0x7fde484a1b20 <unknown>
 #7 0x7fde484ad52b <unknown>
 #8 0x7fde484ad596 <unknown>
 #9 0x7fde484ad7f8 <unknown>
 #10 0x7fde484a1730 <unknown>
 #11 0xa674d1 _ZN11Json_stringC4IJRPcRmEEEDpOT_ at /opt/rh/gcc-toolset-12/root/usr/include/c++/12/bits/new_allocator.h:137
 #12 0xa674d1 _Z14create_dom_ptrI11Json_stringJRPcRmEESt10unique_ptrIT_St14default_deleteIS5_EEDpOT0_ at mysql-8.4.13/sql-common/json_dom.h:140
 #13 0xa674d1 ParseString<0, rapidjson::MemoryStream, (anonymous namespace)::Rapid_json_handler> at mysql-8.4.13/sql-common/json_dom.cc:543
 #14 0xa65d25 ParseObject<0, rapidjson::MemoryStream, (anonymous namespace)::Rapid_json_handler> at mysql-8.4.13/extra/rapidjson/include/rapidjson/reader.h:772
 #15 0xa65d25 ParseValue<0, rapidjson::MemoryStream, (anonymous namespace)::Rapid_json_handler> at mysql-8.4.13/extra/rapidjson/include/rapidjson/reader.h:1758
 #16 0xf35066 Parse<0, rapidjson::MemoryStream, (anonymous namespace)::Rapid_json_handler> at mysql-8.4.13/extra/rapidjson/include/rapidjson/reader.h:575
 #17 0x135ddd5 _ZN10Field_json5storeEPKcmPK12CHARSET_INFO at mysql-8.4.13/sql/field.cc:7692
 #18 0xc27b96 _ZN4Item13save_in_fieldEP5Fieldb at mysql-8.4.13/sql/item.cc:6869
 #19 0xcb16b2 _Z11fill_recordP3THDP5TABLERK14mem_root_dequeIP4ItemES8_P9MY_BITMAPSA_b at mysql-8.4.13/sql/sql_base.cc:9752
 #20 0xcb0ece _Z36fill_record_n_invoke_before_triggersP3THDP9COPY_INFORK14mem_root_dequeIP4ItemES8_P5TABLE23enum_trigger_event_typeibPb at mysql-8.4.13/sql/sql_base.cc:10081
 #21 0xcaf7ae _ZN21Sql_cmd_insert_values13execute_innerEP3THD at mysql-8.4.13/sql/sql_insert.cc:619
 #22 0xca282f _ZN11Sql_cmd_dml7executeEP3THD at mysql-8.4.13/sql/sql_select.cc:782
 #23 0xb4de87 _Z21mysql_execute_commandP3THDb at mysql-8.4.13/sql/sql_parse.cc:4748
 #24 0xd2d262 _Z20dispatch_sql_commandP3THDP12Parser_state at mysql-8.4.13/sql/sql_parse.cc:5417
 #25 0xd24cea _Z16dispatch_commandP3THDPK8COM_DATA19enum_server_command at mysql-8.4.13/sql/sql_parse.cc:2140
 #26 0xd21745 _Z10do_commandP3THD at mysql-8.4.13/sql/sql_parse.cc:1465
 #27 0xd12140 handle_connection at mysql-8.4.13/sql/conn_handler/connection_handler_per_thread.cc:304
 #28 0xfad074 pfs_spawn_thread at mysql-8.4.13/storage/perfschema/pfs.cc:3067
 #29 0x7fde48089d21 <unknown>
 #30 0x7fde4810ed3f <unknown>
 #31 0xffffffffffffffff <unknown>

Trying to get some variables.
Some pointers may be invalid and cause the dump to abort.
Query (7fde19781d70): INSERT INTO test.t (id, p_json) VALUES (FLOOR(RAND()*1000000000), '{\"padding\": \"xxxxxx(...)

How to repeat:
* Use a VM with vm.overcommit_memory=2; 
* Create an example table in MySQL:
CREATE TABLE `t` (
  `id` int NOT NULL,
  `p_json` json DEFAULT NULL,
  PRIMARY KEY (`id`)
) ENGINE=InnoDB;

* Check the current commit limit, i.e.
# awk '/^CommitLimit/{l=$2} /^Committed_AS/{a=$2} END{print int((l-a)/1024)}' /proc/meminfo
10404

* Reserve memory outside of MySQL, leaving a small headroom, i.e., using this simple script:

# cat ballast.py 
#!/usr/bin/env python3
"""Reserve committed (not touched) memory so Committed_AS rises without raising RSS.
usage: ballast.py MB   - prints READY <mb actually reserved> and sleeps until killed."""
import mmap, sys, time

want = int(sys.argv[1])
chunk = 16
held = []
got = 0
while got < want:
    n = min(chunk, want - got)
    try:
        held.append(mmap.mmap(-1, n << 20, flags=mmap.MAP_PRIVATE | mmap.MAP_ANONYMOUS,
                              prot=mmap.PROT_READ | mmap.PROT_WRITE))
    except (OSError, ValueError):
        break                      # commit limit reached
    got += n
print("READY", got, flush=True)
while True:
    time.sleep(3600)

(Try between 200 to 400 MB headroom)
# H=300; python3 /root/ballast.py $((FREE-H))
READY 10104

* Keep the above session holding the mem and verify headroom from another session:
# awk '/^CommitLimit/{l=$2} /^Committed_AS/{a=$2} END{print int((l-a)/1024)}' /proc/meminfo
297

* Now insert around 20MB or bigger JSON object into the p_json column:
mysql --max-allowed-packet=64M < /tmp/insert.sql

Retry a couple of times if needed, and try different headroom values if the crash does not happen immediately.

Suggested fix:
MySQL should ideally be prepared to handle refusals of memory allocation requests. An error message is expected instead of a crash.

The more detailed trace captured with gdb:

(gdb) bt
#0  0x00007fde4808ba6c in __pthread_kill_implementation () from /lib64/libc.so.6
#1  0x00007fde4803e686 in raise () from /lib64/libc.so.6
#2  0x00007fde48028833 in abort () from /lib64/libc.so.6
#3  0x00007fde484a1b21 in __gnu_cxx::__verbose_terminate_handler() [clone .cold] () from /lib64/libstdc++.so.6
#4  0x00007fde484ad52c in __cxxabiv1::__terminate(void (*)()) () from /lib64/libstdc++.so.6
#5  0x00007fde484ad597 in std::terminate() () from /lib64/libstdc++.so.6
#6  0x00007fde484ad7f9 in __cxa_throw () from /lib64/libstdc++.so.6
#7  0x00007fde484a1731 in operator new(unsigned long) [clone .cold] () from /lib64/libstdc++.so.6
#8  0x0000000000a674d2 in std::__new_allocator<char>::allocate (this=<optimized out>, __n=28800001) at /opt/rh/gcc-toolset-12/root/usr/include/c++/12/bits/new_allocator.h:137
#9  std::allocator<char>::allocate (__n=28800001, this=0x7fde19107d60) at /opt/rh/gcc-toolset-12/root/usr/include/c++/12/bits/allocator.h:188
#10 std::allocator_traits<std::allocator<char> >::allocate (__n=28800001, __a=...) at /opt/rh/gcc-toolset-12/root/usr/include/c++/12/bits/alloc_traits.h:464
#11 std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::_M_create (this=0x7fde19107d60, __old_capacity=0, __capacity=<synthetic pointer>: <optimized out>) at /opt/rh/gcc-toolset-12/root/usr/include/c++/12/bits/basic_string.tcc:155
#12 std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::_M_construct<char*> (__end=<optimized out>, __beg=0xa971c20 'x' <repeats 200 times>..., this=0x7fde19107d60) at /opt/rh/gcc-toolset-12/root/usr/include/c++/12/bits/basic_string.tcc:225
#13 std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::basic_string (__a=..., __n=28800000, __s=0xa971c20 'x' <repeats 200 times>..., this=0x7fde19107d60) at /opt/rh/gcc-toolset-12/root/usr/include/c++/12/bits/basic_string.h:620
#14 Json_string::Json_string<char*&, unsigned long&> (this=0x7fde19107d50) at ../../mysql-8.4.13/sql-common/json_dom.h:706
#15 create_dom_ptr<Json_string, char*&, unsigned long&> () at ../../mysql-8.4.13/sql-common/json_dom.h:140
#16 (anonymous namespace)::Rapid_json_handler::String (length=28800000, str=0xa971c20 'x' <repeats 200 times>..., this=0x7fde306f3130) at ../../mysql-8.4.13/sql-common/json_dom.cc:543
#17 rapidjson::GenericReader<rapidjson::UTF8<char>, rapidjson::UTF8<char>, rapidjson::CrtAllocator>::ParseString<0, rapidjson::MemoryStream, (anonymous namespace)::Rapid_json_handler> (this=<optimized out>, is=..., handler=..., isKey=<optimized out>)
    at ../../mysql-8.4.13/extra/rapidjson/include/rapidjson/reader.h:982
#18 0x0000000000a65d26 in rapidjson::GenericReader<rapidjson::UTF8<char>, rapidjson::UTF8<char>, rapidjson::CrtAllocator>::ParseObject<0, rapidjson::MemoryStream, (anonymous namespace)::Rapid_json_handler> (handler=..., is=..., this=0x7fde306f30c0)
    at ../../mysql-8.4.13/extra/rapidjson/include/rapidjson/reader.h:770
#19 rapidjson::GenericReader<rapidjson::UTF8<char>, rapidjson::UTF8<char>, rapidjson::CrtAllocator>::ParseValue<0, rapidjson::MemoryStream, (anonymous namespace)::Rapid_json_handler> (this=0x7fde306f30c0, is=..., handler=...)
    at ../../mysql-8.4.13/extra/rapidjson/include/rapidjson/reader.h:1758
#20 0x0000000000f35067 in rapidjson::GenericReader<rapidjson::UTF8<char>, rapidjson::UTF8<char>, rapidjson::CrtAllocator>::Parse<0, rapidjson::MemoryStream, (anonymous namespace)::Rapid_json_handler> (handler=..., is=..., this=0x7fde306f30c0)
    at ../../mysql-8.4.13/extra/rapidjson/include/rapidjson/reader.h:575
#21 Json_dom::parse (text=<optimized out>, length=28800015, error_handler=..., depth_handler=...) at ../../mysql-8.4.13/sql-common/json_dom.cc:613
#22 0x000000000135ddd6 in Field_json::store (this=0x7fde18eba700, from=<optimized out>, length=<optimized out>, cs=<optimized out>) at ../../mysql-8.4.13/sql/field.cc:7692
#23 0x0000000000c27b97 in Item::save_in_field (this=0x7fde18efe520, field=0x7fde18eba700, no_conversions=<optimized out>) at ../../mysql-8.4.13/sql/item.cc:6869
#24 0x0000000000cb16b3 in fill_record (thd=thd@entry=0x7fde18ece1b0, table=table@entry=0x7fde19248030, fields=..., values=..., bitmap=bitmap@entry=0x0, insert_into_fields_bitmap=insert_into_fields_bitmap@entry=0x0, raise_autoinc_has_expl_non_null_val=<optimized out>)
    at ../../mysql-8.4.13/sql/sql_base.cc:9752
#25 0x0000000000cb0ecf in fill_record_n_invoke_before_triggers (thd=thd@entry=0x7fde18ece1b0, optype_info=optype_info@entry=0x7fde306f35f0, fields=..., values=..., table=table@entry=0x7fde19248030, event=event@entry=TRG_EVENT_INSERT, num_fields=2,
    raise_autoinc_has_expl_non_null_val=true, is_row_changed=0x0) at ../../mysql-8.4.13/sql/sql_base.cc:10081
#26 0x0000000000caf7af in Sql_cmd_insert_values::execute_inner (this=0x7fde1908da10, thd=0x7fde18ece1b0) at ../../mysql-8.4.13/sql/sql_insert.cc:619
#27 0x0000000000ca2830 in Sql_cmd_dml::execute (this=0x7fde1908da10, thd=0x7fde18ece1b0) at ../../mysql-8.4.13/sql/sql_select.cc:782
#28 0x0000000000b4de88 in mysql_execute_command (thd=<optimized out>, first_level=<optimized out>) at ../../mysql-8.4.13/sql/sql_parse.cc:4748
#29 0x0000000000d2d263 in dispatch_sql_command (thd=0x7fde18ece1b0, parser_state=<optimized out>) at ../../mysql-8.4.13/sql/sql_parse.cc:5417
#30 0x0000000000d24ceb in dispatch_command (thd=thd@entry=0x7fde18ece1b0, com_data=com_data@entry=0x7fde306f5490, command=COM_QUERY) at ../../mysql-8.4.13/sql/sql_parse.cc:2140
#31 0x0000000000d21746 in do_command (thd=0x7fde18ece1b0) at ../../mysql-8.4.13/sql/sql_parse.cc:1465
#32 0x0000000000d12141 in handle_connection (arg=arg@entry=0x8d4ff40) at ../../mysql-8.4.13/sql/conn_handler/connection_handler_per_thread.cc:304
#33 0x0000000000fad075 in pfs_spawn_thread (arg=0x8df7fb0) at ../storage/perfschema/../../../mysql-8.4.13/storage/perfschema/pfs.cc:3067
#34 0x00007fde48089d22 in start_thread () from /lib64/libc.so.6
#35 0x00007fde4810ed40 in clone3 () from /lib64/libc.so.6

I forgot to add the line to set the FREE variable before calling the ballast script:

# FREE=$(awk '/^CommitLimit/{l=$2} /^Committed_AS/{a=$2} END{print int((l-a)/1024)}' /proc/meminfo); echo "free commit now: ${FREE} MB, ballast: $((FREE-H)) MB"