Bug #29201 decimal(20, 20) has rounding error

• Submitted: 19 Jun 2007 8:45
• Modified: 2 Jul 2007 4:48
• Reporter: Hakan Küçükyılmaz
• Status: Closed
• Category: MySQL Server: Falcon storage engine
• Severity: S2 (Serious)
• Version: 6.0.1-alpha
• OS: Any
• Assigned to: Christopher Powers
• CPU Architecture: Any

[19 Jun 2007 8:45] Hakan Küçükyılmaz

Description:
Selecting a decimal value shows rounding errors with Falcon.

How to repeat:
[10:44] root@test>CREATE TABLE t1 (a decimal(20, 20)) Engine Falcon;
Query OK, 0 rows affected (0.01 sec)

[10:44] root@test>INSERT INTO t1 VALUES (0.123456789012345678901234567890);
Query OK, 1 row affected, 1 warning (0.00 sec)

[10:44] root@test>INSERT INTO t1 VALUES (0.123456789);
Query OK, 1 row affected (0.01 sec)

[10:44] root@test>SELECT a FROM t1;
+------------------------+
| a                      |
+------------------------+
| 0.00000000000000000000 |
| 0.00000000000000000000 |
+------------------------+
2 rows in set (0.00 sec)

6.0.1-alpha-debug

[19 Jun 2007 8:47] Hakan Küçükyılmaz

Test case is falcon_bug_29201.test

[19 Jun 2007 18:10] MySQL Verification Team

Thank you for the bug report. Verified as described.

[20 Jun 2007 2:42] Christopher Powers

This was caused by bitshift overflows in BigInt::divide(). The overflows are now prevented by casting the intermediate results of the bitshift operations to a 64-bit integer.

[20 Jun 2007 4:20] Hakan Küçükyılmaz

falcon_bug_29201.test passes now.

[2 Jul 2007 4:48] MC Brown

A note has been added to the 6.0.1 changelog:

Wide DECIMAL columns would show rounding errors during SELECT. (Bug#29201)