Bug #62827 MyISAM repair-by-sort buffer cannot be > 4 GB even on 64-bit architectures
Submitted: 19 Oct 2011 18:14 Modified: 19 Oct 2011 18:53
Reporter: Alexey Kopytov Email Updates:
Status: Duplicate Impact on me:
Category:MySQL Server: MyISAM storage engine Severity:S3 (Non-critical)
Version:5.1, 5.5 OS:Any
Assigned to: CPU Architecture:Any
Triage: Needs Triage: D3 (Medium)

[19 Oct 2011 18:14] Alexey Kopytov
This applies to both server's myisam_sort_buffer_size and myisamchk
--sort_buffer_size. Even though both options are declared as GET_ULONG
which theoretically allows them to be greater than 4 GB on 64-bit
architectures (with 64-bit Windows being an exception), their value is
eventually cast to uint in _create_index_by_sort().

As a result, a wrap-around value is actually used as a buffer size. When
the wrap-around value is 0, _create_index_by_sort() fails with a
spurious "myisam_sort_buffer_size is too small" error.

How to repeat:
Try using myisam_sort_buffer_size=4G or "myisamchk -r --sort_buffer_size=4G" on a 64-bit machine.

Suggested fix:
Remove casts to uint in _create_index_by_sort() and calls to it. Better yet, declare the options as GET_ULL with SIZE_T_MAX as their max allowed value, and change all relevant variables from ulong to ulonglong to fix 64-bit Windows as well.
[19 Oct 2011 18:27] Valeriy Kravchuk
Thank you for the bug report. Verified by code review of storage/myisam/sort.c (with current mysql-5.5).
[19 Oct 2011 18:48] Shane Bester
seen bug #45702 ?
[19 Oct 2011 18:53] Valeriy Kravchuk
Indeed, bug #45702 explicitly mentions the following:

"in the code below we get parameter as  ulong and then cast it again to uint

int _create_index_by_sort(MI_SORT_PARAM *info,my_bool no_messages,
                          ulong sortbuff_size)
 uint memavl,old_memavl,keys,sort_length;

among other things, and this is the code this report is about it seems. We can call this a duplicate.