Bug #94560 record comparison in spatial index non-leaf rtree node seems incorrect
Submitted: 5 Mar 2019 11:43 Modified: 7 Mar 2019 4:29
Reporter: Jie Zhou Email Updates:
Status: Verified Impact on me:
None 
Category:MySQL Server: GIS Severity:S2 (Serious)
Version:5.7, 8.0 OS:Any
Assigned to: CPU Architecture:Any
Tags: gis, spatial index

[5 Mar 2019 11:43] Jie Zhou
Description:
According to the function cmp_rec_rec_with_match:
---------
	for (; cur_field < rec1_n_fields && cur_field < rec2_n_fields;
	     cur_field++) {

		ulint	mtype;
		ulint	prtype;

		/* If this is node-ptr records then avoid comparing node-ptr
		field. Only key field needs to be compared. */
		if (cur_field == dict_index_get_n_unique_in_tree(index)) {
			break;
		}

		...
			const dict_col_t*	col;

			col	= dict_index_get_nth_col(index, cur_field);

			mtype = col->mtype;
			prtype = col->prtype;

			/* If the index is spatial index, we mark the
			prtype of the first field as MBR field. */
			if (cur_field == 0 && dict_index_is_spatial(index)) {
				ut_ad(DATA_GEOMETRY_MTYPE(mtype));
				prtype |= DATA_GIS_MBR;
			}
		}
                 ...
		ret = cmp_data(mtype, prtype,
			       rec1_b_ptr, rec1_f_len,
			       rec2_b_ptr, rec2_f_len);
		if (ret) {
			goto order_resolved;
		}
	}
---------
If the record is located in a non-leaf node of spatial index, it consists of two fields, mbr and node-ptr.

"if (cur_field == 0 && dict_index_is_spatial(index)) prtype |= DATA_GIS_MBR;" 
It only deals with the first col.

For the second field, mtype will be set to col->mtype.
It equals to index->fields[1]->col->mtype, which is the primary key's mtype.
So that we compare node-ptr field by pk's mtype.
It is ok in most cases since pk is always set to a int type in real world.
But if pk is varchar type or others, some unexpected problem may happen.

How to repeat:
Currently, I just read the code direcly and find the problem.

Suggested fix:
if (dict_index_is_spatial(index)
    && (!page_is_leaf(page_align(rec1)) || !page_is_leaf(page_align(rec2)))) {
    cmp_spatial_index_nonleaf = true;
}
if (cur_field == 1 && cmp_spatial_index_nonleaf) {
    /* mark as node-ptr */
    mtype = DATA_SYS_CHILD;
    prtype = DATA_NOT_NULL;
}
[7 Mar 2019 4:29] MySQL Verification Team
Hi,

Thanks for your report and suggested fix.

all best
Bogdan