Bug #9708 calculation error

• Submitted: 7 Apr 2005 9:13
• Modified: 7 Apr 2005 12:12
• Reporter: Oliver Peters
• Status: Not a Bug
• Category: MySQL Server
• Severity: S3 (Non-critical)
• Version: 4.1.11
• OS: Windows (Windows2000)
• Assigned to: Sergei Golubchik
• CPU Architecture: Any

[7 Apr 2005 9:13] Oliver Peters

Description:
SELECT 2.1/0.3*1.000000000000000
-> 7.000000000000001

SELECT 2.1/0.3*1.0000000000000000
-> 7.0000000000000009

In my opinion this is an error. 2.1/0.3 has to be exactly 7. I really rely on this believe

How to repeat:
see description

Suggested fix:
I only know a little awk and perl - so I'm almost helpless. In this case "totally helpless" ;-)

[7 Apr 2005 12:12] Sergei Golubchik

Well, as you know Perl, you can see that problem is intrinsic deficiency of floating point numbers - they are not precise:

$ perl -e 'printf "%.15f\n", 2.1/0.3'
7.000000000000001
$ perl -e 'printf "%.16f\n", 2.1/0.3'
7.0000000000000009

But MySQL 5.0.3 uses a precision math library that performs arithmetic calculatiions exactly, without precision loss.

[7 Apr 2005 12:41] Oliver Peters

thank you very much - now I understand and can sleep better

Oliver Peters