Bug #73964 unhex() runtime error: left shift of negative value -1

• Submitted: 18 Sep 2014 7:27
• Modified: 21 Jan 2016 22:49
• Reporter: Shane Bester (Platinum Quality Contributor)
• Status: Closed
• Category: MySQL Server: Data Types
• Severity: S3 (Non-critical)
• Version: 5.7.6
• OS: Any (x64)
• CPU Architecture: Any
• Tags: asan, shift, UNHEX

[18 Sep 2014 7:27] Shane Bester

Description:
On asan build:

/mysql-trunk-clean/sql/item_strfunc.cc:4755:46: runtime error: left shift of negative value -1

That is function String *Item_func_unhex::val_str(String *str)
---
  for (end=res->ptr()+res->length(); from < end ; from+=2, to++)
  {
    int hex_char;
    *to= (hex_char= hexchar_to_int(from[0])) << 4;
    if ((null_value= (hex_char == -1)))
      return 0;
---

How to repeat:
make a build with clang and -DWITH_ASAN=ON:
execute the sql:

select unhex('5078-04-25');

Suggested fix:
left shifting negative values is considered undefined behaviour:

http://www.open-std.org/JTC1/SC22/WG14/www/docs/n1256.pdf

"The result of E1 << E2 is E1 left-shifted E2 bit positions; vacated bits are filled with
zeros. If E1 has an unsigned type, the value of the result is E1 ´ 2E2, reduced modulo
one more than the maximum value representable in the result type. If E1 has a signed
type and nonnegative value, and E1 ´ 2E2 is representable in the result type, then that is
the resulting value; otherwise, the behavior is undefined."

[21 Sep 2014 18:58] MySQL Verification Team

simple fix:

=== modified file 'sql/item_strfunc.cc'
--- sql/item_strfunc.cc 2014-09-12 14:42:43 +0000
+++ sql/item_strfunc.cc 2014-09-21 17:47:43 +0000
@@ -4752,9 +4752,10 @@
   for (end=res->ptr()+res->length(); from < end ; from+=2, to++)
   {
     int hex_char;
-    *to= (hex_char= hexchar_to_int(from[0])) << 4;
+    hex_char= hexchar_to_int(from[0]);
     if ((null_value= (hex_char == -1)))
       return 0;
+    *to= hex_char << 4;
     *to|= hex_char= hexchar_to_int(from[1]);
     if ((null_value= (hex_char == -1)))
       return 0;

[21 Jan 2016 22:49] Paul DuBois

Noted in 5.7.12, 5.8.0 changelogs.

UNHEX() could attempt a left shift by a negative number of bits.

[21 Jan 2016 22:51] Paul DuBois

Correction: UNHEX() could attempt a left shift of a negative number.