Bug #8451 Precision math: Division result incorrect with large decimals

• Submitted: 11 Feb 2005 18:54
• Modified: 8 Jun 2005 10:32
• Reporter: Trudy Pelzer
• Status: Closed
• Category: MySQL Server
• Severity: S3 (Non-critical)
• Version: 5.0.3-alpha-debug
• OS: Linux (SuSE 9.1)
• Assigned to: Alexey Botchkov
• CPU Architecture: Any

[11 Feb 2005 18:54] Trudy Pelzer

Description:
Division with a large decimal number sometimes
returns the wrong answer. The cut-off appears
to be when the precision of the number exceeds
36 digits -- that is, a number with 36 digits divided
by 10 gets the right result, while a number with 37
digits gets the wrong result when divided by 10.

How to repeat:
mysql> create table t1 (col1 int, col2 decimal(38));
Query OK, 0 rows affected (0.01 sec)

mysql> insert into t1 values(1,123456789012345678901234567890123456);
Query OK, 1 row affected (0.00 sec)

mysql> insert into t1 values(2,1234567890123456789012345678901234567);
Query OK, 1 row affected (0.00 sec)

mysql> select col2/10 from t1 where col1=1;
+-------------------------------------------+
| col2/10                                   |
+-------------------------------------------+
| 12345678901234567890123456789012345.60000 |
+-------------------------------------------+
1 row in set (0.00 sec)
-- This is the correct result.

mysql> select col2/10 from t1 where col1=2;
+--------------------------------------------+
| col2/10                                    |
+--------------------------------------------+
| 123456789000000000012345678901234567.89012 |
+--------------------------------------------+
1 row in set (0.00 sec)
-- When one more digit is added to the value, though,
the division operation stops working. The correct result
is: 123456789012345678901234567890123456.7