MySQL Bugs: #68697: loose rec_per_key precision according to whole integer type

Bug #68697	loose rec_per_key precision according to whole integer type
Submitted:	17 Mar 2013 13:56	Modified:	17 Mar 2013 14:20
Reporter:	Oleg Tsarev	Email Updates:
Status:	Verified	Impact on me:	None
Category:	MySQL Server: Storage Engine API	Severity:	S5 (Performance)
Version:	5.5.31, 5.6.11	OS:	Any
Assigned to:		CPU Architecture:	Any
Triage:	Needs Triage: D3 (Medium)

Description:
./storage/innobase/handler/ha_innodb.c: innodb_rec_per_key(
...
{
    ha_rows		rec_per_key;

....
            rec_per_key = (ha_rows)(
                (records - num_null)
                / (index->stat_n_diff_key_vals[i + 1]
                   - num_null));
        }
    } else {
        rec_per_key = (ha_rows)
             (records / index->stat_n_diff_key_vals[i + 1]);
    }

    return(rec_per_key);
}

./include/my_base.h:typedef ulong		ha_rows;

How to repeat:
Execute any query

Suggested fix:
Change type to double

Thank you for the report.

Verified as described.

In version 5.6 code is modified, but it has same rounding issue. This can lead to up to 49.99% deviation when optimizer calculates number of unique rows and lead to choice of wrong indexes.