Bug #117113 MySQL Shell should report problematic strings when running util.checkForServerUpgrade()
Submitted: 7 Jan 7:03 Modified: 7 Jan 8:01
Reporter: Hai Nguyen Xuan Email Updates:
Status: Verified Impact on me:
None 
Category:Shell Upgrade Checker Severity:S2 (Serious)
Version:8.4.x, 8.0.40 OS:Any
Assigned to: CPU Architecture:Any

[7 Jan 7:03] Hai Nguyen Xuan
Description:
Hi team,

In MySQL version 5.7.44, it's possible to insert binary ASCII strings into a field with CHARACTER SET ascii COLLATE ascii_bin. However, in MySQL version 8.0.36, the same insert fails with the error "ERROR 1366 (HY000): Incorrect string value: '\xAE\x1F\xA4B\x8F\xC3...' for column 'col_id' at row 1".

MySQL Shell should detect these problematic strings when running util.checkForServerUpgrade()

How to repeat:
- Create example table in MySQL version 5.7.44:

CREATE TABLE `test`.`example` (
  `col_id` varchar(26) CHARACTER SET ascii COLLATE ascii_bin NOT NULL DEFAULT '',
  `col_date` int(10) unsigned NOT NULL,
  `col_varchar_1` varchar(26) NOT NULL DEFAULT '',
  `col_varchar_2` varchar(26) NOT NULL DEFAULT '',
  `col_varchar_3` varchar(26) NOT NULL DEFAULT '',
  `col_varchar_4` varchar(26) NOT NULL DEFAULT '',
  `col_varchar_5` varchar(26) NOT NULL DEFAULT '',
  `col_varchar_6` varchar(26) NOT NULL DEFAULT '',
  `col_varchar_7` varchar(26) NOT NULL DEFAULT '',
  `col_varchar_8` varchar(26) NOT NULL DEFAULT '',
  `col_varchar_9` varchar(26) NOT NULL DEFAULT '',
  `col_varchar_10` varchar(26) NOT NULL DEFAULT '',
  PRIMARY KEY (`col_id`,`col_date`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 ROW_FORMAT=COMPRESSED;

- Bash script to insert binary ASCII strings:

function client() {
    rows="";
    for i in {1..500}; do {
        col_id_value=$(dd if=/dev/urandom bs=1 count=26 2>/dev/null | tr -d "'" | xxd  -c26 -ps);
        col_date_value=$(shuf -i1686356775-1692255600 -n1);
        rows="(BINARY UNHEX('${col_id_value}'),${col_date_value},'col_varchar_value','col_varchar_value','col_varchar_value','col_varchar_value','col_varchar_value','col_varchar_value','col_varchar_value','col_varchar_value','col_varchar_value','col_varchar_value'),${rows}";
    } done;

    echo "SET sql_mode=''; INSERT INTO test.example (col_id,col_date,col_varchar_1,col_varchar_2,col_varchar_3,col_varchar_4,col_varchar_5,col_varchar_6,col_varchar_7,col_varchar_8,col_varchar_9,col_varchar_10) VALUES ${rows%,}" | mysql -P5744;
}

threads_max=8;
while true; do {
     r=$(jobs -r | wc -l);
     [[ $r == 0 ]] && r=1;
     if [[ $r < $threads_max ]]; then {
         for thread in $(seq ${r} ${threads_max}); do {
             echo "[$(date +%F_%T)] Starting new client.";
             client &
        } done;
    } fi;
    sleep 1;
    inserted=$(mysql -P5744 -BNe "SHOW GLOBAL STATUS LIKE 'Innodb_rows_inserted'" 2>/dev/null| awk '{print $2}');
    echo "[$(date +%F_%T)] Inserted: $((inserted-inserted_prev))";
    inserted_prev=$inserted;
} done;

- Example value in the table:

mysql> select * from example limit 1 \G
*************************** 1. row ***************************
        col_id:  ???*Vf?j?T{?l?4?g??%???
      col_date: 1691623832
 col_varchar_1: col_varchar_value
 col_varchar_2: col_varchar_value
 col_varchar_3: col_varchar_value
 col_varchar_4: col_varchar_value
 col_varchar_5: col_varchar_value
 col_varchar_6: col_varchar_value
 col_varchar_7: col_varchar_value
 col_varchar_8: col_varchar_value
 col_varchar_9: col_varchar_value
col_varchar_10: col_varchar_value
1 row in set (0.00 sec)

- Run checkForServerUpgrade using MySQL Shell version 8.4.0:

mysqlsh -- util check-for-server-upgrade { --user=root --host=127.0.0.1 --port=5744 } --target-version=8.0.36 --config-path=/home/hai.nguyen/sandboxes/msb_5_7_44/my.sandbox.cnf

- Currently, checkForServerUpgrade doesn't detect problematic binary ASCII strings

$ mysqlsh -- util check-for-server-upgrade { --user=root --host=127.0.0.1 --port=5744 } --target-version=8.0.36 --config-path=/home/hai.nguyen/sandboxes/msb_5_7_44/my.sandbox.cnf
Please provide the password for 'root@127.0.0.1:5744': ****
Save password for 'root@127.0.0.1:5744'? [Y]es/[N]o/Ne[v]er (default No):
The MySQL server at 127.0.0.1:5744, version 5.7.44 - MySQL Community Server
(GPL), will now be checked for compatibility issues for upgrade to MySQL 8.0.36.
1) Usage of old temporal type (oldTemporal)
  No issues found
2) MySQL syntax check for routine-like objects (routineSyntax)
  No issues found
3) Usage of db objects with names conflicting with new reserved keywords
(reservedKeywords)
  No issues found
4) Usage of utf8mb3 charset (utf8mb3)
  No issues found
5) Table names in the mysql schema conflicting with new tables in the latest
MySQL. (mysqlSchema)
  No issues found
6) Partitioned tables using engines with non native partitioning
(nonNativePartitioning)
  No issues found
7) Foreign key constraint names longer than 64 characters (foreignKeyLength)
  No issues found
8) Usage of obsolete MAXDB sql_mode flag (maxdbSqlModeFlags)
  No issues found
9) Usage of obsolete sql_mode flags (obsoleteSqlModeFlags)
  Notice: The following DB objects have obsolete options persisted for
    sql_mode, which will be cleared during the upgrade.
  More information:
    https://dev.mysql.com/doc/refman/8.0/en/mysql-nutshell.html#mysql-nutshell-removals
  @@global.sql_mode - defined using obsolete NO_AUTO_CREATE_USER option
10) ENUM/SET column definitions containing elements longer than 255 characters
(enumSetElementLength)
  No issues found
11) Usage of partitioned tables in shared tablespaces
(partitionedTablesInSharedTablespaces)
  No issues found
12) Circular directory references in tablespace data file paths
(circularDirectory)
  No issues found
13) Usage of removed functions (removedFunctions)
  No issues found
14) Usage of removed GROUP BY ASC/DESC syntax (groupbyAscSyntax)
  No issues found
15) Removed system variables for error logging to the system log configuration
(removedSysLogVars)
  No issues found
16) Removed system variables (removedSysVars)
  No issues found
17) System variables with new default values (sysVarsNewDefaults)
  Warning: Following system variables that are not defined in your
    configuration file will have new default values. Please review if you rely on
    their current values and if so define them before performing upgrade.
  More information:
    https://dev.mysql.com/blog-archive/new-defaults-in-mysql-8-0/
  back_log - default value will change.
  character_set_server - default value will change from latin1 to utf8mb4.
  collation_server - default value will change from latin1_swedish_ci to
    utf8mb4_0900_ai_ci.
  event_scheduler - default value will change from OFF to ON.
  explicit_defaults_for_timestamp - default value will change from OFF to ON.
  innodb_autoinc_lock_mode - default value will change from 1 (consecutive) to
    2 (interleaved).
  innodb_flush_method - default value will change from NULL to fsync (Unix),
    unbuffered (Windows).
  innodb_flush_neighbors - default value will change from 1 (enable) to 0
    (disable).
  innodb_max_dirty_pages_pct - default value will change from 75 (%)  90 (%).
  innodb_max_dirty_pages_pct_lwm - default value will change from_0 (%) to 10
    (%).
  innodb_undo_log_truncate - default value will change from OFF to ON.
  innodb_undo_tablespaces - default value will change from 0 to 2.
  log_bin - default value will change from OFF to ON.
  log_error_verbosity - default value will change from 3 (Notes) to 2 (Warning).
  log_slave_updates - default value will change from OFF to ON.
  master_info_repository - default value will change from FILE to TABLE.
  max_allowed_packet - default value will change from 4194304 (4MB) to 67108864
    (64MB).
  max_error_count - default value will change from 64 to 1024.
  optimizer_trace_max_mem_size - default value will change from 16KB to 1MB.
  performance_schema_consumer_events_transactions_current - default value will
    change from OFF to ON.
  performance_schema_consumer_events_transactions_history - default value will
    change from OFF to ON.
  relay_log_info_repository - default value will change from FILE to TABLE.
  server_id - default value will change from 0 to 1.
  slave_rows_search_algorithms - default value will change from 'INDEX_SCAN,
    TABLE_SCAN' to 'INDEX_SCAN, HASH_SCAN'.
  table_open_cache - default value will change from 2000 to 4000.
  transaction_write_set_extraction - default value will change from OFF to
    XXHASH64.
18) Zero Date, Datetime, and Timestamp values (zeroDates)
  No issues found
19) Schema inconsistencies resulting from file removal or corruption
(schemaInconsistency)
  No issues found
20) Tables recognized by InnoDB that belong to a different engine (engineMixup)
  No issues found
21) Issues reported by 'check table x for upgrade' command (checkTableCommand)
  No issues found
22) New default authentication plugin considerations
(defaultAuthenticationPlugin)
  Warning: The new default authentication plugin 'caching_sha2_password' offers
    more secure password hashing than previously used 'mysql_native_password'
    (and consequent improved client connection authentication). However, it also
    has compatibility implications that may affect existing MySQL installations.
    If your MySQL installation must serve pre-8.0 clients and you encounter
    compatibility issues after upgrading, the simplest way to address those
    issues is to reconfigure the server to revert to the previous default
    authentication plugin (mysql_native_password). For example, use these lines
    in the server option file:
    [mysqld]
    default_authentication_plugin=mysql_native_password
    However, the setting should be viewed as temporary, not as a long term or
    permanent solution, because it causes new accounts created with the setting
    in effect to forego the improved authentication security.
    If you are using replication please take time to understand how the
    authentication plugin changes may impact you.
  More information:
    https://dev.mysql.com/doc/refman/8.0/en/upgrading-from-previous-series.html#upgrade-cachin...
    https://dev.mysql.com/doc/refman/8.0/en/upgrading-from-previous-series.html#upgrade-cachin...
23) Indexes on functions with changed semantics
(changedFunctionsInGeneratedColumns)
  No issues found
24) Columns which cannot have default values (columnsWhichCannotHaveDefaults)
  No issues found
25) Check for invalid table names and schema names used in 5.7 (invalid57Names)
  No issues found
26) Check for orphaned routines and events in 5.7 (orphanedObjects)
  No issues found
27) Check for deprecated usage of single dollar signs in object names
(dollarSignName)
  No issues found
28) Check for indexes that are too large to work on higher versions of MySQL
Server than 5.7 (indexTooLarge)
  No issues found
29) Check for deprecated '.<table>' syntax used in routines.
(emptyDotTableSyntax)
  No issues found
30) Check for columns that have foreign keys pointing to tables from a
different database engine. (invalidEngineForeignKey)
  No issues found
31) Check for deprecated or invalid user authentication methods.
(authMethodUsage)
  Warning: The following users are using the 'mysql_native_password'
  authentication method which is deprecated as of MySQL 8.0.0 and will be
  removed in a future release.
  Consider switching the users to a different authentication method (i.e.
  caching_sha2_password).
  - msandbox@127.%
  - msandbox@localhost
  - msandbox_ro@127.%
  - msandbox_ro@localhost
  - msandbox_rw@127.%
  - msandbox_rw@localhost
  - mysql.session@localhost
  - mysql.sys@localhost
  - root@localhost
  - rsandbox@127.%
  More information:
    https://dev.mysql.com/doc/refman/8.0/en/caching-sha2-pluggable-authentication.html
32) Check for deprecated or removed plugin usage. (pluginUsage)
  No issues found
33) Check for deprecated or invalid default authentication methods in system
variables. (deprecatedDefaultAuth)
  The following variables have problems with their set authentication method:
  Warning: default_authentication_plugin - mysql_native_password authentication
    method is deprecated and it should be considered to correct this before
    upgrading to 8.4.0 release.
34) Check for deprecated or invalid authentication methods in use by MySQL
Router internal accounts. (deprecatedRouterAuthMethod)
  No issues found
35) Check for deprecated temporal delimiters in table partitions.
(deprecatedTemporalDelimiter)
  No issues found
Errors:   0
Warnings: 38
Notices:  1
NOTE: No fatal errors were found that would prevent an upgrade, but some potential issues were detected. Please ensure that the reported issues are not significant before upgrading.

- Upgrade MySQL Server from 5.7.44 to 8.0.36:

2025-01-07T01:50:53.375740Z mysqld_safe Starting mysqld daemon with databases from /home/hai.nguyen/sandboxes/msb_8_0_36/data
2025-01-07T01:50:53.784862Z 0 [System] [MY-010116] [Server] /home/hai.nguyen/mysql/8.0.36/bin/mysqld (mysqld 8.0.36) starting as process 2811352
2025-01-07T01:50:53.809761Z 1 [System] [MY-011012] [Server] Starting upgrade of data directory.
2025-01-07T01:50:53.809850Z 1 [System] [MY-013576] [InnoDB] InnoDB initialization has started.
2025-01-07T01:50:54.883351Z 1 [System] [MY-013577] [InnoDB] InnoDB initialization has ended.
2025-01-07T01:50:56.819970Z 2 [System] [MY-011003] [Server] Finished populating Data Dictionary tables with data.
2025-01-07T01:50:58.349064Z 5 [System] [MY-013381] [Server] Server upgrade from '50700' to '80036' started.
2025-01-07T01:51:07.175504Z 5 [System] [MY-013381] [Server] Server upgrade from '50700' to '80036' completed.
2025-01-07T01:51:07.265191Z 0 [Warning] [MY-010068] [Server] CA certificate ca.pem is self signed.
2025-01-07T01:51:07.265258Z 0 [System] [MY-013602] [Server] Channel mysql_main configured to support TLS. Encrypted connections are now supported for this channel.
2025-01-07T01:51:07.299297Z 0 [System] [MY-011323] [Server] X Plugin ready for connections. Bind-address: '::' port: 18036, socket: /tmp/mysqlx-18036.sock
2025-01-07T01:51:07.299438Z 0 [System] [MY-010931] [Server] /home/hai.nguyen/mysql/8.0.36/bin/mysqld: ready for connections. Version: '8.0.36'  socket: '/tmp/mysql_sandbox8036.sock'  port: 8036  MySQL Community Server - GPL.

- Create identical table and insert value from table example will fail, showing that these values cannot be inserted in MySQL 8.x:

mysql> create table example2 as select * from example;
ERROR 1366 (HY000): Incorrect string value: '\xAE\x1F\xA4B\x8F\xC3...' for column 'col_id' at row 1
[7 Jan 8:01] MySQL Verification Team
Hello Hai Nguyen Xuan,

Thank you for the report and feedback.
Verified as described.

regards,
Umesh