Bug #24607 MyISAM pointer size determined incorrectly
Submitted: 27 Nov 2006 6:52 Modified: 7 Feb 2007 12:16
Reporter: Yoshiaki Tajika (Basic Quality Contributor) Email Updates:
Status: Closed Impact on me:
None 
Category:MySQL Server: MyISAM storage engine Severity:S2 (Serious)
Version:5.0.32-BK, 5.0.27 OS:Linux (Linux)
Assigned to: Ingo Strüwing CPU Architecture:Any

[27 Nov 2006 6:52] Yoshiaki Tajika
Description:
MyISAM pointer size is determined incorrectly,
when avg_row_length * max_rows >= 256TB.

How to repeat:
You can easily notice a wrong code at mi_get_pointer_length().
See below.

Suggested fix:
myisam/mi_create.c
mi_get_pointer_length()

---------------------------------------------------
#ifdef NOT_YET_READY_FOR_8_BYTE_POINTERS
    if (file_length >= (longlong) 1 << 56)
      def=8;
#endif
    if (file_length >= (longlong) 1 << 48)  /* (a) */
      def=7;
    if (file_length >= (longlong) 1 << 40)  /* (b) */
      def=6;
    else if (file_length >= (longlong) 1 << 32)
      def=5;
---------------------------------------------------

IF-block (b) should be "else if", otherwise `def` never becomes 7.
Besides, in case NOT_YET_READY_FOR_8_BYTE_POINTERS is defined,
IF-block (a) also is a bug.
[27 Nov 2006 8:13] Valeriy Kravchuk
Thank you for a bug report. In current 5.0.32-BK sources we still have:

uint mi_get_pointer_length(ulonglong file_length, uint def)
{
  DBUG_ASSERT(def >= 2 && def <= 7);
  if (file_length)                              /* If not default */
  {
#ifdef NOT_YET_READY_FOR_8_BYTE_POINTERS
    if (file_length >= (longlong) 1 << 56)
      def=8;
#endif
    if (file_length >= (longlong) 1 << 48)
      def=7;
    if (file_length >= (longlong) 1 << 40)
      def=6;
    else if (file_length >= (longlong) 1 << 32)
      def=5;
    else if (file_length >= (1L << 24))
      def=4;
    else if (file_length >= (1L << 16))
      def=3;
    else
      def=2;
  }
  return def;
}

So, there is obvious bug for 2 cases you noted.
[30 Dec 2006 21:55] Ingo Strüwing
Hi.

I will fix this bug. But it will change nothing. Still you will not be able to create tables >= 256T by using MAX_ROWS and AVG_ROW_LENGTH. The problem is that the .frm file has only four bytes for MAX_ROWS. So it is limited to 4G-1. Rows can have a maximum length of 64K-1. So the product is somewhere below 256T.

The only way to get tables bigger than this is to SET GLOBAL myisam_data_pointer_size= 7 and *not* to specify MAX_ROWS and AVG_ROW_LENGTH.

If you want to raise the limit of an existing table, do SET GLOBAL myisam_data_pointer_size= 7 and ALTER TABLE table_name MAX_ROWS= 0. A pointer size of 7 allows for data files up to 65535T. MAX_ROWS= 0 means unlimited row count. The rows are counted with 8-byte variables so it is possible to go beyond the highest possible limit.

You can check the success of this operation with SHOW TABLE STATUS LIKE 'table_name'.

Regards
Ingo
[2 Jan 2007 14:55] Bugs System
A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/17523

ChangeSet@1.2590, 2007-01-02 15:55:06+01:00, istruewing@chilla.local +1 -0
  Bug#24607 - MyISAM pointer size determined incorrectly
  
  The function mi_get_pointer_length() computed too small
  pointer size for very large tables.
  
  Inserted missing 'else' between the branches for very
  large tables.
  
  No test case. The effect is not easy to show. When creating
  a table with AVG_ROW_LENGTH/MAX_ROWS, file_length cannot be
  big enough. MAX_ROWS is limited by a 4-byte space in the .frm
  file. Hence file_length is limited to < 256T.
  
  The result of mi_get_pointer_length() is used internally in
  myisampack and in the server for sorting and optimizing.
  These would require an existing table >= 256T to provoke an
  failure due to the incorrectly computed pointer length.
[5 Jan 2007 2:49] Yoshiaki Tajika
Hi, according to the previous comment below,

 >  These would require an existing table >= 256T to provoke an
 >  failure due to the incorrectly computed pointer length.

do you need some correction of code anywhere other than 
mi_get_pointer_length()?
[5 Jan 2007 9:19] Ingo Strüwing
Hi, no, I don't think so. Unless one wants to fix the limitation of MAX_ROWS. But this could introduce all sorts of incompatibilities with former versions.

Meanwhile I noticed that my explanations regarding the impossibility of creating tables >= 256T was wrong.

I said that records cannot be bigger than 64K-1 bytes. This is wrong for records with blobs. The limit exists only for the sum of the non-blob parts of a record, including the blob length field and an in-memory pointer. The blob data itself don't count for the record length limit.

Consequentially one can specify AVG_ROW_LENGTH > 65535. I will commit a new patch with a test case soon.

Regards
Ingo
[5 Jan 2007 9:27] Bugs System
A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/17655

ChangeSet@1.2590, 2007-01-05 10:26:51+01:00, istruewing@chilla.local +3 -0
  Bug#24607 - MyISAM pointer size determined incorrectly
  
  The function mi_get_pointer_length() computed too small
  pointer size for very large tables.
  
  Inserted missing 'else' between the branches for very
  large tables.
[24 Jan 2007 12:17] Bugs System
A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/18698

ChangeSet@1.2593, 2007-01-24 13:17:01+01:00, istruewing@chilla.local +2 -0
  Bug#24607 - MyISAM pointer size determined incorrectly
  Fixed test. On 32-bit machines which compile without
  -DBIG_TABLES, MAX_ROWS is truncated to a 32-bit value.
  Using a value below 4G is portable.
[24 Jan 2007 12:30] Bugs System
A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/18700

ChangeSet@1.2363, 2007-01-24 13:29:46+01:00, istruewing@chilla.local +1 -0
  Bug#24607 - MyISAM pointer size determined incorrectly
  After merge fix
[25 Jan 2007 8:23] Ingo Strüwing
Queued to 5.1-engines, 5.0-engines, 4.1-engines.
[25 Jan 2007 9:30] Sergey Vojtovich
Pushed to trees currently marked as 5.0.36 and 5.1.15.
[5 Feb 2007 11:42] Sergey Vojtovich
Pushed to tree currently marked as 4.1.23.
[7 Feb 2007 12:16] MC Brown
Closed. No documentation needed, and no additions to the changelog have been made.