Bug #88337 mod result differs between hex and decimal values

• Submitted: 2 Nov 2017 17:52
• Modified: 3 Nov 2017 10:05
• Reporter: Bill McCaffrey
• Status: Verified
• Category: MySQL Server: Parser
• Severity: S3 (Non-critical)
• Version: 5.5.53-0ubuntu0.14.04.1-log
• OS: Ubuntu (14.04.5)
• CPU Architecture: Any

[2 Nov 2017 17:52] Bill McCaffrey

Description:
Doing mod operations on certain large values gives a different result if the operand is in hex notation or decimal.

How to repeat:
mysql> select 13845242653677231%0x7FFFFFFF;
+------------------------------+
| 13845242653677231%0x7FFFFFFF |
+------------------------------+
|                   1117124361 |
+------------------------------+
1 row in set (0.00 sec)

mysql> select 13845242653677231%2147483647;
+------------------------------+
| 13845242653677231%2147483647 |
+------------------------------+
|                   1117124360 |
+------------------------------+
1 row in set (0.00 sec)

[3 Nov 2017 10:05] MySQL Verification Team

Hello Bill McCaffrey,

Thank you for the report and test case.
Observed that 5.6.38, 5.5.57 are affected, 5.7.20 seems not affected.

Thanks,
Umesh

[3 Nov 2017 10:08] MySQL Verification Team

-- 5.7.20

root@localhost [(none)]> select 13845242653677231%0x7FFFFFFF;
+------------------------------+
| 13845242653677231%0x7FFFFFFF |
+------------------------------+
|                   1117124360 |
+------------------------------+
1 row in set (0.00 sec)

root@localhost [(none)]> select 13845242653677231%2147483647;
+------------------------------+
| 13845242653677231%2147483647 |
+------------------------------+
|                   1117124360 |
+------------------------------+
1 row in set (0.00 sec)

-- 5.6.38, 5.5.57, 5.1.77 and 5.0.96

root@localhost [(none)]> select 13845242653677231%0x7FFFFFFF;
+------------------------------+
| 13845242653677231%0x7FFFFFFF |
+------------------------------+
|                   1117124361 |
+------------------------------+
1 row in set (0.00 sec)

root@localhost [(none)]> select 13845242653677231%2147483647;
+------------------------------+
| 13845242653677231%2147483647 |
+------------------------------+
|                   1117124360 |
+------------------------------+
1 row in set (0.00 sec)