Bug #121236 TINYINT UNSIGNED values corrupted (negative or 18446744073709551615) when read via SSPS
Submitted: 5 Sep 4:57
Reporter: Munechika Machida Email Updates:
Status: Open Impact on me:
None 
Category:Connector / ODBC Severity:S2 (Serious)
Version:26.7.1 OS:Other (Amazon Linux 2023,RHEL9)
Assigned to: CPU Architecture:x86

[5 Sep 4:57] Munechika Machida
Description:
I have not tested or verified a fix myself — this is based on reading
the source, so parts may be wrong. Feedback welcome.

Problem: TINYINT UNSIGNED columns with values above 127 come back
corrupted — either negative, or as 18446744073709551615
(0xFFFFFFFFFFFFFFFF). Reproduces with a plain SELECT, no WHERE clause
or parameters needed, as long as the statement goes through
SQLPrepare()+SQLExecute() (SQLExecDirect() does not trigger it,
see below).

Likely cause: driver/my_prepared_stmt.cc, function ssps_get_int64().
In the 1-byte (TINYINT) case, the UNSIGNED branch reads the buffer as
plain (char *) instead of (unsigned char *):

    case 1:
      if (is_it_unsigned)
      {
        ret = !is_it_null? ((char *)col_rbind->buffer)[0]:0;  // <-- here
      }

Since char is signed on most platforms, a byte like 0xFF (255) is read
as -1. When the template return type T is unsigned long long, that -1
gets sign-extended into 18446744073709551615. The 2/4/8-byte cases in
the same function correctly use unsigned pointer types when
is_it_unsigned is true — only the 1-byte case doesn't.

Why it always reproduces regardless of query shape: since commit
d304fd9f (Bug #32079486, 2021-02-15), SQLPrepare() always forces SSPS
on (force_prepare=true), even without a '?' in the query.
SQLExecDirect() still only uses SSPS when the query has a '?'. Most
client libraries call SQLPrepare()+SQLExecute() for everything, which
is presumably why this shows up unconditionally in practice.

This (char *) cast itself dates back to the first SSPS implementation
(commit 049d4701, 2012-03-15). I could not reproduce it on 5.2.5-8;
my guess is that's simply because SQLPrepare() didn't force SSPS back
then. Not verified.

How to repeat:
1. CREATE TABLE t (id INT PRIMARY KEY, flag TINYINT UNSIGNED);
   INSERT INTO t VALUES (1, 255);

2. Execute via SQLPrepare() + SQLExecute() (not SQLExecDirect()):

   SELECT id, flag FROM t;

3. Bind "flag" with SQL_C_SBIGINT or SQL_C_UBIGINT.

4. Result is -1 or 18446744073709551615 instead of 255.

Suggested fix:
Not tested — just a guess based on the pattern used elsewhere in the
same function.

File:     driver/my_prepared_stmt.cc
Function: template <typename T> T ssps_get_int64(STMT *stmt, ulong column_number, char *value, ulong length)

    case 1:
      if (is_it_unsigned)
      {
-       ret = !is_it_null? ((char *)col_rbind->buffer)[0]:0;
+       ret = !is_it_null? ((unsigned char *)col_rbind->buffer)[0]:0;
      }
      else
      {
        ret = !is_it_null? *(char *)(col_rbind->buffer):0;
      }
      break;