MySQL Bugs: #62827: MyISAM repair-by-sort buffer cannot be

Bug #62827	MyISAM repair-by-sort buffer cannot be > 4 GB even on 64-bit architectures
Submitted:	19 Oct 2011 18:14	Modified:	19 Oct 2011 18:53
Reporter:	Alexey Kopytov	Email Updates:
Status:	Duplicate	Impact on me:	None
Category:	MySQL Server: MyISAM storage engine	Severity:	S3 (Non-critical)
Version:	5.1, 5.5	OS:	Any
Assigned to:		CPU Architecture:	Any

Description:
This applies to both server's myisam_sort_buffer_size and myisamchk
--sort_buffer_size. Even though both options are declared as GET_ULONG
which theoretically allows them to be greater than 4 GB on 64-bit
architectures (with 64-bit Windows being an exception), their value is
eventually cast to uint in _create_index_by_sort().

As a result, a wrap-around value is actually used as a buffer size. When
the wrap-around value is 0, _create_index_by_sort() fails with a
spurious "myisam_sort_buffer_size is too small" error.

How to repeat:
Try using myisam_sort_buffer_size=4G or "myisamchk -r --sort_buffer_size=4G" on a 64-bit machine.

Suggested fix:
Remove casts to uint in _create_index_by_sort() and calls to it. Better yet, declare the options as GET_ULL with SIZE_T_MAX as their max allowed value, and change all relevant variables from ulong to ulonglong to fix 64-bit Windows as well.

Thank you for the bug report. Verified by code review of storage/myisam/sort.c (with current mysql-5.5).

seen bug #45702 ?

Indeed, bug #45702 explicitly mentions the following:

"in the code below we get parameter as  ulong and then cast it again to uint

int _create_index_by_sort(MI_SORT_PARAM *info,my_bool no_messages,
                          ulong sortbuff_size)
{
...
 uint memavl,old_memavl,keys,sort_length;
...
 memavl=max(sortbuff_size,MIN_SORT_MEMORY);
...
}
"

among other things, and this is the code this report is about it seems. We can call this a duplicate.