Bug #121010 Contribution by ByteDance RDS MySQL team: mysqld crashes on keyring_key_store UDF return-type mismatch

• Submitted: 27 Jul 7:15
• Reporter: alex xing (OCA)
• Status: Open
• Category: MySQL Server: User-defined functions ( UDF )
• Severity: S2 (Serious)
• Version: 8.4.7-7
• OS: Any
• CPU Architecture: Any

[27 Jul 7:15] alex xing

Description:
  The legacy UDF loader allows keyring_key_store to be created with an SQL return type that does not match the C UDF implementation. keyring_key_store is implemented as an integer-returning
  UDF, but the following statement succeeds:

  CREATE FUNCTION keyring_key_store RETURNS STRING SONAME 'keyring_udf.so';

  When a real keyring backend is installed and the key store operation succeeds, the C UDF returns integer 1. However, because the SQL layer believes the UDF returns STRING, the result is
  interpreted as a string pointer. The integer value 1 is then treated as an invalid address, and mysqld crashes while sending the result to the client.

  This does not reproduce when no keyring backend is available, because the UDF fails before returning success. It reproduces when keyring_udf is installed and a keyring backend such as
  component_keyring_file or keyring_file is active.

  Steps to reproduce:

  DROP FUNCTION IF EXISTS keyring_key_store;
  CREATE FUNCTION keyring_key_store RETURNS STRING SONAME 'keyring_udf.so';

  4. Execute a successful store:

  SELECT keyring_key_store('wrong_return_type_aes_key', 'AES', '0123456789abcdef');

  Actual result:
  The client loses the connection and mysqld crashes with SIGSEGV.

  Example stack frames:

  Protocol_classic::store_string(...)
  Item::send(...)
  THD::send_result_set_row(...)
  Query_result_send::send_data(...)
  Query_expression::ExecuteIteratorQuery(...)
  mysql_execute_command(...)
  dispatch_sql_command(...)

  Expected result:
  mysqld should not crash. The server should reject the call safely, ideally during UDF initialization, with an error similar to:

  ERROR HY000: Can't initialize function 'keyring_key_store'; Return type mismatch: keyring_key_store must be declared with RETURNS INTEGER.

  The failed call should not store the key, and the client connection should remain usable.

  Root cause:
  keyring_key_store has an integer C ABI, but CREATE FUNCTION accepts an arbitrary SQL RETURNS type for the same SONAME symbol. If the SQL declaration says RETURNS STRING, the server
  dispatches/sends the UDF result as a string. A successful keyring_key_store returns integer 1, which is misinterpreted as a char pointer, leading to a crash in result sending.

  Impact:
  A user who can create/install UDFs can crash mysqld when a working keyring backend is available. This is a server availability issue caused by missing validation between the SQL-declared
  UDF return type and the actual keyring UDF return type.

How to repeat:
mysql-test/suite/component_keyring_file/t/udf_wrong_return_type_crash.test : 

--source include/not_windows.inc
--source include/have_component_keyring_file.inc

# keyring_udf wrong return type safety regression:
# The SQL layer declares keyring_key_store as RETURNS STRING even though the
# underlying C UDF returns INTEGER. The call must be rejected before key
# storage runs, and the connection must remain usable.

--source ../inc/setup_component.inc

--echo # Install keyring_udf plugin and declare keyring_key_store with the wrong SQL return type.
--replace_regex /\.dll/.so/
--eval INSTALL PLUGIN keyring_udf SONAME '$KEYRING_UDF'
--replace_regex /\.dll/.so/
--eval CREATE FUNCTION keyring_key_store RETURNS STRING SONAME '$KEYRING_UDF'
--replace_regex /\.dll/.so/
--eval CREATE FUNCTION keyring_key_fetch RETURNS STRING SONAME '$KEYRING_UDF'
--replace_regex /\.dll/.so/
--eval CREATE FUNCTION keyring_key_remove RETURNS INTEGER SONAME '$KEYRING_UDF'

--echo # The incorrect declaration is rejected before the keyring store runs.
--error ER_CANT_INITIALIZE_UDF
SELECT keyring_key_store(
  'wrong_return_type_aes_key', 'AES', '0123456789abcdef');

--echo # The same connection remains usable and the rejected call stored no key.
SELECT 1 AS server_is_alive;
SELECT keyring_key_fetch('wrong_return_type_aes_key') IS NULL AS key_not_stored;

--echo # Recreate keyring_key_store with its correct return type.
DROP FUNCTION keyring_key_store;
--replace_regex /\.dll/.so/
--eval CREATE FUNCTION keyring_key_store RETURNS INTEGER SONAME '$KEYRING_UDF'

SELECT keyring_key_store(
  'wrong_return_type_aes_key', 'AES', '0123456789abcdef');
SELECT LENGTH(keyring_key_fetch('wrong_return_type_aes_key'));
SELECT keyring_key_remove('wrong_return_type_aes_key');

--echo # Clean-up
DROP FUNCTION keyring_key_store;
DROP FUNCTION keyring_key_fetch;
DROP FUNCTION keyring_key_remove;
UNINSTALL PLUGIN keyring_udf;

--source ../inc/teardown_component.inc

Suggested fix:
  Validate the SQL-declared return type for keyring_key_store during UDF initialization.

  keyring_key_store is implemented as an integer-returning UDF, so its init function should reject any declaration whose return type is not INTEGER/INT_RESULT. The failure should happen
  before the store operation runs, returning ER_CANT_INITIALIZE_UDF instead of allowing the result to be sent through the wrong UDF ABI.

  A minimal fix is to expose the already available declared UDF result type to the keyring_udf plugin, then add a keyring_key_store-specific check in keyring_key_store_init():

  - If declared return type is INT_RESULT, continue normally.
  - Otherwise, set an error message such as:
    "Return type mismatch: keyring_key_store must be declared with RETURNS INTEGER."
  - Return initialization failure.
  - Do not execute the key store operation.

  This preserves CREATE FUNCTION compatibility while preventing the SELECT-time crash and avoiding any keyring side effect from the rejected call.