Bug #69715 UBSAN: Item_func_mul::int_op() mishandles 9223372036854775809*-1
Submitted: 10 Jul 2013 17:39 Modified: 3 Sep 22:19
Reporter: Arthur O'Dwyer Email Updates:
Status: Closed Impact on me:
None 
Category:MySQL Server: Data Types Severity:S3 (Non-critical)
Version: OS:Any
Assigned to: CPU Architecture:Any
Tags: ubsan

[10 Jul 2013 17:39] Arthur O'Dwyer
Description:
mysqld  Ver 5.5.31-0ubuntu0.12.04.2 for debian-linux-gnu on x86_64 ((Ubuntu))

MySQL believes correctly that 9223372036854775809 is of type BIGINT UNSIGNED. But if you ask MySQL to compute 9223372036854775809*-1, it will return the BIGINT UNSIGNED value 9223372036854775807 instead of throwing an arithmetic-overflow error.

How to repeat:
SELECT 9223372036854775808 * -1;
SELECT 9223372036854775809 * -1;

The first query above correctly yields:
    ERROR 1690 (22003): BIGINT UNSIGNED value is out of range in '(9223372036854775808 * -(1))'

The second query incorrectly yields:
    +--------------------------+
    |      9223372036854775807 |
    +--------------------------+

Suggested fix:
The bug appears to be due to a typo in Item_func_mul::int_op(). It was part of the original fix for Bug #8433, which added all the overflow-checking code.
https://github.com/twitter/mysql/commit/197260313f6093a06675854a35bb62f3834d0879

    if ((ulonglong) res > (ulonglong) LONGLONG_MIN + 1)
      goto err;

I believe the proper fix is to change that "LONGLONG_MIN" to "LONGLONG_MAX". (Compare the similar code in Item_func_neg and Item_func_abs, which already correctly uses LONGLONG_MAX.)

You should also add a regression test to "mysql-test/t/func_math.test".
[10 Jul 2013 17:53] Arthur O'Dwyer
Here is a slightly less contrived and therefore slightly more frightening test case; I recommend adding it as a regression test.

create table ints (a bigint unsigned, b bigint signed);
insert into ints values (119537721,-77158673929);
select a*b from ints;

    +---------------------+
    | a*b                 |
    +---------------------+
    | 9223372036854775807 |
    +---------------------+
[10 Jul 2013 18:47] Sinisa Milivojevic
Indeed, this is what I get with 5.6.12:

[sinisa@local mysql-5.6.12]$ ./client/Debug/mysql bug -e "SELECT 9223372036854775808 * -1;"
ERROR 1690 (22003) at line 1: BIGINT UNSIGNED value is out of range in '(9223372036854775808 * -(1))'
[sinisa@local mysql-5.6.12]$ ./client/Debug/mysql bug -e "SELECT 9223372036854775809 * -1;"
+--------------------------+
| 9223372036854775809 * -1 |
+--------------------------+
|      9223372036854775807 |
+--------------------------+
[23 Feb 2014 8:28] Shane Bester
mysql-trunk built with clang shows us:

./mysql-trunk-clean/sql/item_func.cc:1612:10: runtime error: negation of -9223372036854775808 cannot be represented in type 'longlong' (aka 'long long'); cast to an unsigned type to negate this value to itself
[23 Feb 2014 20:53] Arthur O'Dwyer
@Shane: You only tried the first line, though; you didn't try the second line nor the "slightly less contrived and therefore slightly more frightening test case" in my first comment. IIRC, clang -fsanitize won't catch THAT bug.

It's good that you guys are finally using -fsanitize, though. You'll find a ton of bugs that way. :)
[30 Aug 2014 18:11] Shane Bester
Seen again in mysql-trunk with function longlong Item_func_neg::int_op().
Just run mysqld built with ASAN and execute this:

select -(-9223372036854775808), -9223372036854775808;

./mysql-trunk-clean/sql/item_func.cc:2164:33: runtime error: negation of -9223372036854775808 cannot be represented in type 'longlong' (aka 'long long'); cast to an unsigned type to negate this value to itself

./mysql-trunk-clean/strings/decimal.c:1028:20: runtime error: negation of -9223372036854775808 cannot be represented in type 'longlong' (aka 'long long'); cast to an unsigned type to negate this value to itself
[21 Oct 2017 8:12] Shane Bester
Still fails on current trunk ubsan build.   The query:  SELECT 9223372036854775808 * -1;

[NOTE] mysqld: Version: '9.0.0-dmr-ubsan' (Built on 20 October 2017)
item_func.cc:2006:8: runtime error: negation of -9223372036854775808 cannot be represented in type 'long long int'; cast to an unsigned type to negate this value to itself
#0 in Item_func_mul::int_op() ./sql/item_func.cc:2006
#1 in Item_func_numhybrid::val_int() ./sql/item_func.cc:1346
#2 in Item::send ./sql/item.cc:7625
#3 in THD::send_result_set_row ./sql/sql_class.cc:2883
#4 in Query_result_send::send_data ./sql/query_result.cc:98
#5 in JOIN::exec() ./sql/sql_executor.cc:238
#6 in Sql_cmd_dml::execute_inner ./sql/sql_select.cc:728
#7 in Sql_cmd_dml::execute ./sql/sql_select.cc:608
#8 in mysql_execute_command ./sql/sql_parse.cc:4642
#9 in mysql_parse ./sql/sql_parse.cc:5435
#10 in dispatch_command ./sql/sql_parse.cc:1713
#11 in do_command(THD*) ./sql/sql_parse.cc:1299
#12 in handle_connection ./connection_handler_per_thread.cc:328
#13 in pfs_spawn_thread ./storage/perfschema/pfs.cc:2987

Code in question in Item_func_mul::int_op()

  if (a_negative != b_negative)
  {
    if ((ulonglong) res > (ulonglong) LLONG_MIN + 1)
      goto err;
    res= -res;    <------------
  }
[3 Sep 22:19] Paul Dubois
Posted by developer:
 
Fixed in 8.0.13.

A range check for the product of a signed and unsigned integer could
be performed incorrectly.