MySQL Bugs: #8576: Test case 'group_min_max' fails on HP-UX and Solaris x86

Bug #8576	Test case 'group_min_max' fails on HP-UX and Solaris x86
Submitted:	17 Feb 2005 16:55	Modified:	3 Mar 2005 18:30
Reporter:	Kent Boortz	Email Updates:
Status:	Closed	Impact on me:	None
Category:	MySQL Server	Severity:	S1 (Critical)
Version:	5.0.3-pre	OS:	HP/UX (HP-UX)
Assigned to:	Timour Katchaounov	CPU Architecture:	Any

Description:
Test case 'group_min_max' fails on HP-UX and Solaris x86

How to repeat:
Run the test case on the specified targets.

Some effects:

=== on hp3750 in 'classic', 'max', 'pro', 'standard': ===
group_min_max                  [ fail ]

Errors are (from ...) :
.../mysqltest: At line 347: query 'select a1,a2,b,       max(c) from t1 where (c < 'a0') group by a1,a2,b' failed: 2013: Lost connection to MySQL server during query
(the last lines may be the most important ones)

Ending Tests
Shutting-down MySQL daemon

master not cooperating with mysqladmin, will try manual kill
master refused to die. Sending SIGKILL
Master shutdown finished
Slave shutdown finished
=== end hp3750 ===

Same effect on 'hpux11', noted only in 32 bit builds, with tendency to hang until manual kill.

As explained by serg, the problem was in different representation of double variables
depending on platform/compiler/optimization. In some cases such variables are
repsented with 80 bits in the FPU, while normally with 64 bits. As a result, seemingly
equal values are not considered equal. The current fix declares such variables as 'volatile'
to force all being represented with 64 bits.

Fixed in 5.0.3.
This bug was in fact consisting of two completely unrelated problems, that it why we have two
separate changesets.
1) the first problem was test failure on Solaris/Intel due to a compiler optimization of doubles.
In this case the compiler used 64 bits for one cost, and 80 bits for for the other. The
comparison of two equal costs resulted in that they are not equal, and as a result the optimizer
chose a different index for several queries, which resulted in failed test case.
2) the second problem was due to a bug in the HPUX C++ compiler. The compiler did
some improper use of registers, and as a result the final call to a loop condition didn't work.
As a result QUICK_GROUP_MIN_MAX_SELECT::get_next() entered into an infinite loop, and
the server had to be killed.

Test suite only. No changelog entry needed.