Bug #68697 loose rec_per_key precision according to whole integer type
Submitted: 17 Mar 2013 13:56 Modified: 17 Mar 2013 14:20
Reporter: Oleg Tsarev Email Updates:
Status: Verified Impact on me:
None 
Category:MySQL Server: Storage Engine API Severity:S5 (Performance)
Version:5.5.31, 5.6.11 OS:Any
Assigned to: CPU Architecture:Any
Triage: Needs Triage: D3 (Medium)

[17 Mar 2013 13:56] Oleg Tsarev
Description:
./storage/innobase/handler/ha_innodb.c: innodb_rec_per_key(
...
{
    ha_rows		rec_per_key;

....
            rec_per_key = (ha_rows)(
                (records - num_null)
                / (index->stat_n_diff_key_vals[i + 1]
                   - num_null));
        }
    } else {
        rec_per_key = (ha_rows)
             (records / index->stat_n_diff_key_vals[i + 1]);
    }

    return(rec_per_key);
}

./include/my_base.h:typedef ulong		ha_rows;

How to repeat:
Execute any query

Suggested fix:
Change type to double
[17 Mar 2013 14:20] Sveta Smirnova
Thank you for the report.

Verified as described.

In version 5.6 code is modified, but it has same rounding issue. This can lead to up to 49.99% deviation when optimizer calculates number of unique rows and lead to choice of wrong indexes.