Bug #120940 check-for-server-upgrade misses removed-function usage when whitespace separates the function name from '('
Submitted: 16 Jul 2:03
Reporter: Chelluru Vidyadhar Email Updates:
Status: Open Impact on me:
None 
Category:Shell Upgrade Checker Severity:S2 (Serious)
Version:8.4.10, 9.7.1 OS:Any
Assigned to: CPU Architecture:Any

[16 Jul 2:03] Chelluru Vidyadhar
Description:
The upgrade checker's `removedFunctions` check (and every check that relies on `mysqlshdk::utils::SQL_iterator::next_sql_function()`) fails to detect usage of a removed function when there is whitespace — a space, tab, or newline — between the function name and its opening parenthesis, for example `PASSWORD ('x')` or `PASSWORD\n('x')` instead of `PASSWORD('x')`.

`func (args)` with intervening whitespace is valid SQL that executes normally under the default `sql_mode` (IGNORE_SPACE is not required). So these are real, working objects that use a removed function.

The miss occurs only for object types whose definitions the server stores verbatim: stored procedures, functions, triggers, and events. Expression-based objects (views, generated columns, functional indexes, CHECK constraints, DEFAULT expressions) are unaffected, because the server rewrites those definitions into a canonical whitespace-free form (`password(...)`) before storing them in information_schema — so those remain correctly detected.

Root cause: 
`next_sql_function()` treats a token as a function call only when the very next character after the name is `(`. It does not skip whitespace between the name and the parenthesis:

    // mysqlshdk/libs/utils/utils_lexing.cc
    std::string_view SQL_iterator::next_sql_function() {
      std::string_view token;
      while (!(token = next_token()).empty()) {
        if (!std::isalpha(token.front())) continue;
        if (valid() && get_char() == '(') break;   // fails when a space is here
      }
      return token;
    }

For the stored text `PASSWORD ('x')`, `next_token()` returns `"PASSWORD"` and leaves the cursor on the space; the `get_char() == '('` test is then false, so the name is discarded and the usage is never reported.

Impact: 
an object such as a stored procedure/function/trigger/event that calls a removed function using the very common `FUNC (args)` formatting style passes the upgrade check, and then fails at runtime after the upgrade (the removed function does not resolve).

How to repeat:
On a MySQL 5.7 server (so the removed `PASSWORD()` function still exists):

    CREATE DATABASE test;
    USE test;

    DELIMITER //
    -- caught today (no space)
    CREATE PROCEDURE p_nospace() BEGIN SELECT PASSWORD('x'); END//
    -- MISSED (space before the paren) -- the bug
    CREATE PROCEDURE p_space()   BEGIN SELECT PASSWORD ('x'); END//
    DELIMITER ;

    -- control: same call in a view; the server normalizes the space away,
    -- so this one is (correctly) caught
    CREATE VIEW v_ws AS SELECT PASSWORD ('x') AS h;

Run the checker:

    mysqlsh --user=... --password=... --socket=/tmp/mysql_sandbox5744.sock \
      -- util check-for-server-upgrade --target-version=8.0.40 \
         --include=removedFunctions

Observed output (only the view is reported; the space-form procedure is missed):

    1) Usage of removed functions (removedFunctions)
       Error: The following DB objects use functions that were removed in the
       latest MySQL version. ...

       test.v_ws - VIEW uses removed function PASSWORD

       More information:
         https://dev.mysql.com/doc/refman/8.0/en/mysql-nutshell.html#mysql-nutshell-removals

    Errors:   1
    Warnings: 0
    Notices:  0

    ERROR: 1 errors were found. ...

Expected: `test.p_nospace`, `test.p_space`, and `test.v_ws` should all be
reported (all three call the removed `PASSWORD()` function). `p_space` is
silently missed solely because of the space before `(`.

You can confirm the space is preserved verbatim in the routine definition:

    SELECT routine_name, routine_definition
    FROM information_schema.routines
    WHERE routine_schema='test';
    -- p_space stores:  BEGIN SELECT PASSWORD ('x'); END   (space kept)
    -- p_nospace stores: BEGIN SELECT PASSWORD('x'); END

Version-independent reproduction at the lexer level (no server needed) — the
following returns an empty string instead of "PASSWORD":

    mysqlshdk::utils::SQL_iterator it("PASSWORD ('x')");
    it.next_sql_function();   // returns "" ; expected "PASSWORD"

Suggested fix:
Skip whitespace between the function-name token and the parenthesis check in
`SQL_iterator::next_sql_function()`:

    std::string_view SQL_iterator::next_sql_function() {
      std::string_view token;
      while (!(token = next_token()).empty()) {
        if (!std::isalpha(token.front())) continue;
        while (valid() && std::isspace(get_char())) ++(*this);  // skip whitespace
        if (valid() && get_char() == '(') break;
      }
      return token;
    }