Bug #71559 Max_data_length has integer overflow on some memory tables
Submitted: 2 Feb 2014 12:29
Reporter: Shane Bester (Platinum Quality Contributor) Email Updates:
Status: Verified Impact on me:
None 
Category:MySQL Server: Memory storage engine Severity:S3 (Non-critical)
Version:5.0.96 OS:Windows
Assigned to: CPU Architecture:Any

[2 Feb 2014 12:29] Shane Bester
Description:
Tested on 5.0, 5.1, 5.5, 5.6, 5.7 on Windows x64.
Linux x64 was okay.

At least on windows 64-bit, the Max_data_length is wrong:

mysql> show table status like 't1'\G
*************************** 1. row ***************************
           Name: t1
         Engine: MEMORY
        Version: 10
     Row_format: Fixed
           Rows: 2
 Avg_row_length: 16
    Data_length: 126984
Max_data_length: 0  <-----------
   Index_length: 0
      Data_free: 0
 Auto_increment: NULL
    Create_time: 2014-02-02 14:26:09
    Update_time: NULL
     Check_time: NULL
      Collation: latin1_swedish_ci
       Checksum: NULL
 Create_options:
        Comment:
1 row in set (0.00 sec)

How to repeat:
set tmp_table_size=1024*1024*1024*18;
set max_heap_table_size=1024*1024*1024*18;
drop table if exists `t1`;
create table `t1` (`a` tinyint,`b` tinyint,`c` double,`d` timestamp,`e` tinyint) engine=memory ;
insert into `t1` values (),();
show table status like 't1'\G
[2 Feb 2014 13:01] MySQL Verification Team
This is the assembly:

stats.max_data_file_length= hp_info.max_records * hp_info.reclength;
000000014020DA8E  mov         eax,dword ptr [rsp+30h]  
000000014020DA92  imul        eax,dword ptr [rsp+48h]  
000000014020DA97  mov         eax,eax  
000000014020DA99  mov         rcx,qword ptr [rsp+90h]  
000000014020DAA1  mov         qword ptr [rcx+48h],rax  

rsp+30h has value 0x30000000 (decimal 805306368)
rsp+48h has value 0x10 (decimal 16)

After imul, the flags are showing overflow:

OV = 1 UP = 0 EI = 1 PL = 0 ZR = 0 AC = 0 PE = 1 CY = 1
[2 Feb 2014 13:09] MySQL Verification Team
I fixed it by casting things to ulonglong:

=== modified file 'storage/heap/ha_heap.cc'
--- storage/heap/ha_heap.cc     2014-01-06 06:02:04 +0000
+++ storage/heap/ha_heap.cc     2014-02-02 13:02:38 +0000
@@ -406,7 +406,7 @@
   stats.mean_rec_length=      hp_info.reclength;
   stats.data_file_length=     hp_info.data_length;
   stats.index_file_length=    hp_info.index_length;
-  stats.max_data_file_length= hp_info.max_records * hp_info.reclength;
+  stats.max_data_file_length= (ulonglong)hp_info.max_records * (ulonglong)hp_info.reclength;
   stats.delete_length=        hp_info.deleted * hp_info.reclength;
   stats.create_time=          (ulong) hp_info.create_time;
   if (flag & HA_STATUS_AUTO)

So now the result is shown correctly:

mysql> show table status like 't1'\G
*************************** 1. row ***************************
           Name: t1
         Engine: MEMORY
        Version: 10
     Row_format: Fixed
           Rows: 2
 Avg_row_length: 16
    Data_length: 126984
Max_data_length: 12884901888 <--------
   Index_length: 0
      Data_free: 0
 Auto_increment: NULL
    Create_time: 2014-02-02 15:05:50
    Update_time: NULL
     Check_time: NULL
      Collation: latin1_swedish_ci
       Checksum: NULL
 Create_options:
        Comment:
1 row in set (41.71 sec)
[2 Feb 2014 13:11] MySQL Verification Team
I'm unsure why the compiler (VS2010) cannot handle this by itself.
[2 Feb 2014 13:27] MySQL Verification Team
The assembly of the working patch is now showing that full RAX is used.

stats.max_data_file_length= (ulonglong)hp_info.max_records * (ulonglong)hp_info.reclength;
000000014039DA8E  mov         eax,dword ptr [rsp+30h]  
000000014039DA92  mov         ecx,dword ptr [rsp+48h]  
000000014039DA96  imul        rax,rcx  
000000014039DA9A  mov         rcx,qword ptr [rsp+90h]  
000000014039DAA2  mov         qword ptr [rcx+48h],rax