Bug #73964 unhex() runtime error: left shift of negative value -1
Submitted: 18 Sep 2014 7:27 Modified: 21 Jan 2016 22:49
Reporter: Shane Bester (Platinum Quality Contributor) Email Updates:
Status: Closed Impact on me:
None 
Category:MySQL Server: Data Types Severity:S3 (Non-critical)
Version:5.7.6 OS:Any (x64)
Assigned to: CPU Architecture:Any
Tags: asan, shift, UNHEX

[18 Sep 2014 7:27] Shane Bester
Description:
On asan build:

/mysql-trunk-clean/sql/item_strfunc.cc:4755:46: runtime error: left shift of negative value -1

That is function String *Item_func_unhex::val_str(String *str)
---
  for (end=res->ptr()+res->length(); from < end ; from+=2, to++)
  {
    int hex_char;
    *to= (hex_char= hexchar_to_int(from[0])) << 4;
    if ((null_value= (hex_char == -1)))
      return 0;
---

How to repeat:
make a build with clang and -DWITH_ASAN=ON:
execute the sql:

select unhex('5078-04-25');

Suggested fix:
left shifting negative values is considered undefined behaviour:

http://www.open-std.org/JTC1/SC22/WG14/www/docs/n1256.pdf

"The result of E1 << E2 is E1 left-shifted E2 bit positions; vacated bits are filled with
zeros. If E1 has an unsigned type, the value of the result is E1 ´ 2E2, reduced modulo
one more than the maximum value representable in the result type. If E1 has a signed
type and nonnegative value, and E1 ´ 2E2 is representable in the result type, then that is
the resulting value; otherwise, the behavior is undefined."
[21 Sep 2014 18:58] MySQL Verification Team
simple fix:

=== modified file 'sql/item_strfunc.cc'
--- sql/item_strfunc.cc 2014-09-12 14:42:43 +0000
+++ sql/item_strfunc.cc 2014-09-21 17:47:43 +0000
@@ -4752,9 +4752,10 @@
   for (end=res->ptr()+res->length(); from < end ; from+=2, to++)
   {
     int hex_char;
-    *to= (hex_char= hexchar_to_int(from[0])) << 4;
+    hex_char= hexchar_to_int(from[0]);
     if ((null_value= (hex_char == -1)))
       return 0;
+    *to= hex_char << 4;
     *to|= hex_char= hexchar_to_int(from[1]);
     if ((null_value= (hex_char == -1)))
       return 0;
[21 Jan 2016 22:49] Paul DuBois
Noted in 5.7.12, 5.8.0 changelogs.

UNHEX() could attempt a left shift by a negative number of bits.
[21 Jan 2016 22:51] Paul DuBois
Correction: UNHEX() could attempt a left shift of a negative number.