Bug #37408 Compressed MyISAM files should not require/use mmap()
Submitted: 14 Jun 2008 9:39 Modified: 12 Mar 2010 17:49
Reporter: Domas Mituzas Email Updates:
Status: Closed Impact on me:
None 
Category:MySQL Server: MyISAM storage engine Severity:S3 (Non-critical)
Version:5.0, 5.1 OS:Any (any unix/linux)
Assigned to: Satya B CPU Architecture:Any
Triage: Triaged: D2 (Serious)

[14 Jun 2008 9:39] Domas Mituzas
Description:
Compressed MyISAM files are opened with mmap(), thus putting at same page priority as other MySQL memory data. 

This is causing major process memory to be swapped out due to default MADV_NORMAL policies. 

How to repeat:
Open compressed MyISAM file

Suggested fix:
use madvise(MADV_DONTNEED) on opened MyISAM files.
[13 Aug 2008 10:53] Domas Mituzas
If we don't fix a feature because it is not under widespread use, its use will just diminish. 

Is compressed myisam deprecated and not supported?
[14 Aug 2008 12:30] Domas Mituzas
Changing synopsis - madvise() doesn't really help - it is onetime operation that flushes all mmap buffers. 

The fix for this problem is not using mmap(), and resorting to standard pread()/pwrite()

Without that, compressed MyISAM tables are not usable in bigger production environments.
[28 Nov 2008 23:35] Shane Bester
also note that due to bug #40634 , all temporary myisam tables are read one row at a time instead of using the read_buffer_size caching. So, there's a huge performance decrease, which I guess applies to compressed tables too
[16 Oct 2009 5:41] Bugs System
A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/87063

2821 Satya B	2009-10-16
      Fix for Bug #37408 - Compressed MyISAM files should not require/use mmap()
                        
      When compressed myisam files are opened, they are always memory mapped
      sometimes causing memory swapping problems.
      
      When we mmap the myisam compressed tables of size greater than the memory 
      available, specially with linux kernels older than 2.6.9, the kswapd0 process 
      utilization is very high consuming 30-40% of the cpu.
      
      With newer linux kernels,we don't have this problem of high cpu consumption and
      this option may not be required.
       
      The option 'myisam_mmap_size' is added to limit the amount of memory used for
      memory mapping of myisam files. This option is not dynamic.
                      
      Also for mmaping, we use system's mmap call directly instead of using mysql's 
      wrapper function my_mmap() which made the implementation of the option 
      'mmap_memory_size' difficult and so myisam level option is introduced.
      
      Note: Testcase is not added and has to be tested manually
     @ myisam/mi_packrec.c
        Fix for Bug #37408 - Compressed MyISAM files should not require/use mmap()
        
        add 'myisam_mmap_size' option which limits the memory available to mmap of 
        myisam files
     @ myisam/mi_static.c
        Fix for Bug #37408 - Compressed MyISAM files should not require/use mmap()
        
        declare 'myisam_mmap_size' variable
     @ myisam/myisamdef.h
        Fix for Bug #37408 - Compressed MyISAM files should not require/use mmap()
        
        declare 'myisam_mmap_size' variable
     @ sql/mysql_priv.h
        Fix for Bug #37408 - Compressed MyISAM files should not require/use mmap()
        
        declare 'myisam_mmap_size' variable
     @ sql/mysqld.cc
        Fix for Bug #37408 - Compressed MyISAM files should not require/use mmap()
        
        add the 'myisam_mmap_size' option
     @ sql/set_var.cc
        Fix for Bug #37408 - Compressed MyISAM files should not require/use mmap()
        
        add the 'myisam_mmap_size' to the SHOW VARIABLES list
[11 Nov 2009 9:51] Bugs System
A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/90067

2845 Satya B	2009-11-11
      Fix for Bug #37408 - Compressed MyISAM files should not require/use mmap()
                        
      When compressed myisam files are opened, they are always memory mapped
      sometimes causing memory swapping problems.
      
      When we mmap the myisam compressed tables of size greater than the memory 
      available, the kswapd0 process utilization is very high consuming 30-40% of 
      the cpu. This happens only with linux kernels olders than 2.6.9
      
      With newer linux kernels, we don't have this problem of high cpu consumption
      and this option may not be required.
       
      The option 'myisam_mmap_size' is added to limit the amount of memory used for
      memory mapping of myisam files. This option is not dynamic.
      
      The default value on 32 bit system is 4294967295 bytes and on 64 bit system it
      is 18446744073709547520 bytes.
                      
      Also for mmaping, we use system's mmap call directly instead of using mysql's 
      wrapper function my_mmap() which made the implementation of the option 
      'mmap_memory_size' difficult and so myisam level option is introduced.
      
      Note: Testcase is not added and has to be tested manually
     @ include/my_global.h
        Fix for Bug #37408 - Compressed MyISAM files should not require/use mmap()
        
        define SIZE_T_MAX
     @ include/myisam.h
        Fix for Bug #37408 - Compressed MyISAM files should not require/use mmap()
        
        declare 'myisam_mmap_size' and 'myisam_mmap_size_temp' variables
     @ myisam/mi_packrec.c
        Fix for Bug #37408 - Compressed MyISAM files should not require/use mmap()
        
        add 'myisam_mmap_size' option which limits the memory available to mmap of 
        myisam files
     @ myisam/mi_static.c
        Fix for Bug #37408 - Compressed MyISAM files should not require/use mmap()
        
        declare 'myisam_mmap_size' and 'myisam_mmap_size_temp' variables
     @ sql/mysqld.cc
        Fix for Bug #37408 - Compressed MyISAM files should not require/use mmap()
        
        add the 'myisam_mmap_size' option
     @ sql/set_var.cc
        Fix for Bug #37408 - Compressed MyISAM files should not require/use mmap()
        
        add the 'myisam_mmap_size' to the SHOW VARIABLES list
[12 Nov 2009 11:32] Bugs System
A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/90206

2845 Satya B	2009-11-12
      Fix for Bug #37408 - Compressed MyISAM files should not require/use mmap()
                        
      When compressed myisam files are opened, they are always memory mapped
      sometimes causing memory swapping problems.
      
      When we mmap the myisam compressed tables of size greater than the memory 
      available, the kswapd0 process utilization is very high consuming 30-40% of 
      the cpu. This happens only with linux kernels olders than 2.6.9
      
      With newer linux kernels, we don't have this problem of high cpu consumption
      and this option may not be required.
       
      The option 'myisam_mmap_size' is added to limit the amount of memory used for
      memory mapping of myisam files. This option is not dynamic.
      
      The default value on 32 bit system is 4294967295 bytes and on 64 bit system it
      is 18446744073709547520 bytes.
                      
      Also for mmaping, we use system's mmap call directly instead of using mysql's 
      wrapper function my_mmap() which made the implementation of the option 
      'mmap_memory_size' difficult and so myisam level option is introduced.
      
      Note: Testcase is not added and has to be tested manually
     @ include/my_global.h
        Fix for Bug #37408 - Compressed MyISAM files should not require/use mmap()
        
        define SIZE_T_MAX
     @ include/myisam.h
        Fix for Bug #37408 - Compressed MyISAM files should not require/use mmap()
        
        declare 'myisam_mmap_size' and 'myisam_mmap_used' variables
     @ myisam/mi_packrec.c
        Fix for Bug #37408 - Compressed MyISAM files should not require/use mmap()
        
        add 'myisam_mmap_size' option which limits the memory available to mmap of 
        myisam files
     @ myisam/mi_static.c
        Fix for Bug #37408 - Compressed MyISAM files should not require/use mmap()
        
        declare 'myisam_mmap_size' and 'myisam_mmap_used' variables
     @ sql/mysqld.cc
        Fix for Bug #37408 - Compressed MyISAM files should not require/use mmap()
        
        add the 'myisam_mmap_size' option
     @ sql/set_var.cc
        Fix for Bug #37408 - Compressed MyISAM files should not require/use mmap()
        
        add the 'myisam_mmap_size' to the SHOW VARIABLES list
[23 Nov 2009 5:39] Bugs System
A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/91249

2849 Satya B	2009-11-23
      Fix for Bug #37408 - Compressed MyISAM files should not require/use mmap()
                        
      When compressed myisam files are opened, they are always memory mapped
      sometimes causing memory swapping problems.
      
      When we mmap the myisam compressed tables of size greater than the memory 
      available, the kswapd0 process utilization is very high consuming 30-40% of 
      the cpu. This happens only with linux kernels older than 2.6.9
      
      With newer linux kernels, we don't have this problem of high cpu consumption
      and this option may not be required.
       
      The option 'myisam_mmap_size' is added to limit the amount of memory used for
      memory mapping of myisam files. This option is not dynamic.
      
      The default value on 32 bit system is 4294967295 bytes and on 64 bit system it
      is 18446744073709547520 bytes.
      
      Note: Testcase only tests the option variable. The actual bug has be to 
      tested manually.
     @ include/my_global.h
        Fix for Bug #37408 - Compressed MyISAM files should not require/use mmap()
        
        define SIZE_T_MAX
     @ include/myisam.h
        Fix for Bug #37408 - Compressed MyISAM files should not require/use mmap()
        
        declare 'myisam_mmap_size' and 'myisam_mmap_used' variables
     @ myisam/mi_packrec.c
        Fix for Bug #37408 - Compressed MyISAM files should not require/use mmap()
        
        add 'myisam_mmap_size' option which limits the memory available to mmap of 
        myisam files
     @ myisam/mi_static.c
        Fix for Bug #37408 - Compressed MyISAM files should not require/use mmap()
        
        declare 'myisam_mmap_size' and 'myisam_mmap_used' variables
     @ mysql-test/r/variables.result
        Fix for Bug #37408 - Compressed MyISAM files should not require/use mmap()
        
        Testcase for BUG#37408 to test the myisam_mmap_size option
     @ mysql-test/t/variables.test
        Fix for Bug #37408 - Compressed MyISAM files should not require/use mmap()
        
        Testcase for BUG#37408 to test the myisam_mmap_size option
     @ sql/mysqld.cc
        Fix for Bug #37408 - Compressed MyISAM files should not require/use mmap()
        
        add the 'myisam_mmap_size' option
     @ sql/set_var.cc
        Fix for Bug #37408 - Compressed MyISAM files should not require/use mmap()
        
        add the 'myisam_mmap_size' to the SHOW VARIABLES list
[25 Nov 2009 12:46] Bugs System
A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/91594

2849 Satya B	2009-11-25
      Fix for Bug #37408 - Compressed MyISAM files should not require/use mmap()
                        
      When compressed myisam files are opened, they are always memory mapped
      sometimes causing memory swapping problems.
      
      When we mmap the myisam compressed tables of size greater than the memory 
      available, the kswapd0 process utilization is very high consuming 30-40% of 
      the cpu. This happens only with linux kernels older than 2.6.9
      
      With newer linux kernels, we don't have this problem of high cpu consumption
      and this option may not be required.
       
      The option 'myisam_mmap_size' is added to limit the amount of memory used for
      memory mapping of myisam files. This option is not dynamic.
      
      The default value on 32 bit system is 4294967295 bytes and on 64 bit system it
      is 18446744073709547520 bytes.
      
      Note: Testcase only tests the option variable. The actual bug has be to 
      tested manually.
     @ include/my_global.h
        Fix for Bug #37408 - Compressed MyISAM files should not require/use mmap()
        
        define SIZE_T_MAX
     @ include/myisam.h
        Fix for Bug #37408 - Compressed MyISAM files should not require/use mmap()
        
        declare 'myisam_mmap_size' and 'myisam_mmap_used' variables and the mutex
        THR_LOCK_myisam_mmap
     @ myisam/mi_packrec.c
        Fix for Bug #37408 - Compressed MyISAM files should not require/use mmap()
        
        add 'myisam_mmap_size' option which limits the memory available to mmap of 
        myisam files
     @ myisam/mi_static.c
        Fix for Bug #37408 - Compressed MyISAM files should not require/use mmap()
        
        declare 'myisam_mmap_size' and 'myisam_mmap_used' variables and the mutex
        THR_LOCK_myisam_mmap
     @ mysql-test/r/variables.result
        Fix for Bug #37408 - Compressed MyISAM files should not require/use mmap()
        
        Testcase for BUG#37408 to test the myisam_mmap_size option
     @ mysql-test/t/variables.test
        Fix for Bug #37408 - Compressed MyISAM files should not require/use mmap()
        
        Testcase for BUG#37408 to test the myisam_mmap_size option
     @ sql/mysqld.cc
        Fix for Bug #37408 - Compressed MyISAM files should not require/use mmap()
        
        add the 'myisam_mmap_size' option
     @ sql/set_var.cc
        Fix for Bug #37408 - Compressed MyISAM files should not require/use mmap()
        
        add the 'myisam_mmap_size' to the SHOW VARIABLES list
     @ sql/set_var.h
        Fix for Bug #37408 - Compressed MyISAM files should not require/use mmap()
        
        Add system variable type 'sys_var_ulonglong_const_ptr'
[9 Dec 2009 12:35] Bugs System
A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/93322

2859 Satya B	2009-12-09
      Fix for Bug#37408 - Compressed MyISAM files should not require/use mmap()
                        
      When compressed myisam files are opened, they are always memory mapped
      sometimes causing memory swapping problems.
      
      When we mmap the myisam compressed tables of size greater than the memory 
      available, the kswapd0 process utilization is very high consuming 30-40% of 
      the cpu. This happens only with linux kernels older than 2.6.9
      
      With newer linux kernels, we don't have this problem of high cpu consumption
      and this option may not be required.
       
      The option 'myisam_mmap_size' is added to limit the amount of memory used for
      memory mapping of myisam files. This option is not dynamic.
      
      The default value on 32 bit system is 4294967295 bytes and on 64 bit system it
      is 18446744073709547520 bytes.
      
      Note: Testcase only tests the option variable. The actual bug has be to 
      tested manually.
     @ include/my_global.h
        Fix for Bug #37408 - Compressed MyISAM files should not require/use mmap()
        
        define SIZE_T_MAX
     @ include/myisam.h
        Fix for Bug #37408 - Compressed MyISAM files should not require/use mmap()
        
        declare 'myisam_mmap_size' and 'myisam_mmap_used' variables and the mutex
        THR_LOCK_myisam_mmap
     @ myisam/mi_packrec.c
        Fix for Bug #37408 - Compressed MyISAM files should not require/use mmap()
        
        add 'myisam_mmap_size' option which limits the memory available to mmap of 
        myisam files
     @ myisam/mi_static.c
        Fix for Bug #37408 - Compressed MyISAM files should not require/use mmap()
        
        declare 'myisam_mmap_size' and 'myisam_mmap_used' variables and the mutex
        THR_LOCK_myisam_mmap
     @ mysql-test/r/variables.result
        Fix for Bug #37408 - Compressed MyISAM files should not require/use mmap()
        
        Testcase for BUG#37408 to test the myisam_mmap_size option
     @ mysql-test/t/variables.test
        Fix for Bug #37408 - Compressed MyISAM files should not require/use mmap()
        
        Testcase for BUG#37408 to test the myisam_mmap_size option
     @ mysys/my_thr_init.c
        Fix for Bug #37408 - Compressed MyISAM files should not require/use mmap()
        
        intialize the lock THR_LOCK_myisam_mmap
     @ sql/mysqld.cc
        Fix for Bug #37408 - Compressed MyISAM files should not require/use mmap()
        
        add the 'myisam_mmap_size' option
     @ sql/set_var.cc
        Fix for Bug #37408 - Compressed MyISAM files should not require/use mmap()
        
        add the 'myisam_mmap_size' to the SHOW VARIABLES list
[9 Dec 2009 12:49] Bugs System
A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/93324

2861 Satya B	2009-12-09
      Fix for Bug#37408 - Compressed MyISAM files should not require/use mmap()
                        
      When compressed myisam files are opened, they are always memory mapped
      sometimes causing memory swapping problems.
      
      When we mmap the myisam compressed tables of size greater than the memory 
      available, the kswapd0 process utilization is very high consuming 30-40% of 
      the cpu. This happens only with linux kernels older than 2.6.9
      
      With newer linux kernels, we don't have this problem of high cpu consumption
      and this option may not be required.
       
      The option 'myisam_mmap_size' is added to limit the amount of memory used for
      memory mapping of myisam files. This option is not dynamic.
      
      The default value on 32 bit system is 4294967295 bytes and on 64 bit system it
      is 18446744073709547520 bytes.
      
      Note: Testcase only tests the option variable. The actual bug has be to 
      tested manually.
     @ include/my_global.h
        Fix for Bug #37408 - Compressed MyISAM files should not require/use mmap()
        
        define SIZE_T_MAX
     @ include/myisam.h
        Fix for Bug #37408 - Compressed MyISAM files should not require/use mmap()
        
        declare 'myisam_mmap_size' and 'myisam_mmap_used' variables and the mutex
        THR_LOCK_myisam_mmap
     @ myisam/mi_packrec.c
        Fix for Bug #37408 - Compressed MyISAM files should not require/use mmap()
        
        add 'myisam_mmap_size' option which limits the memory available to mmap of 
        myisam files
     @ myisam/mi_static.c
        Fix for Bug #37408 - Compressed MyISAM files should not require/use mmap()
        
        declare 'myisam_mmap_size' and 'myisam_mmap_used' variables and the mutex
        THR_LOCK_myisam_mmap
     @ mysql-test/r/variables.result
        Fix for Bug #37408 - Compressed MyISAM files should not require/use mmap()
        
        Testcase for BUG#37408 to test the myisam_mmap_size option
     @ mysql-test/t/variables.test
        Fix for Bug #37408 - Compressed MyISAM files should not require/use mmap()
        
        Testcase for BUG#37408 to test the myisam_mmap_size option
     @ mysys/my_thr_init.c
        Fix for Bug #37408 - Compressed MyISAM files should not require/use mmap()
        
        intialize the lock THR_LOCK_myisam_mmap
     @ sql/mysqld.cc
        Fix for Bug #37408 - Compressed MyISAM files should not require/use mmap()
        
        add the 'myisam_mmap_size' option
     @ sql/set_var.cc
        Fix for Bug #37408 - Compressed MyISAM files should not require/use mmap()
        
        add the 'myisam_mmap_size' to the SHOW VARIABLES list
[16 Dec 2009 12:02] Bugs System
A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/94474

2822 Satya B	2009-12-16
      Fix for Bug#37408 - Compressed MyISAM files should not require/use mmap()
                        
      When compressed myisam files are opened, they are always memory mapped
      sometimes causing memory swapping problems.
      
      When we mmap the myisam compressed tables of size greater than the memory 
      available, the kswapd0 process utilization is very high consuming 30-40% of 
      the cpu. This happens only with linux kernels older than 2.6.9
      
      With newer linux kernels, we don't have this problem of high cpu consumption
      and this option may not be required.
       
      The option 'myisam_mmap_size' is added to limit the amount of memory used for
      memory mapping of myisam files. This option is not dynamic.
      
      The default value on 32 bit system is 4294967295 bytes and on 64 bit system it
      is 18446744073709547520 bytes.
      
      Note: Testcase only tests the option variable. The actual bug has be to 
      tested manually.
     @ include/my_global.h
        Fix for Bug #37408 - Compressed MyISAM files should not require/use mmap()
        
        define SIZE_T_MAX
     @ include/myisam.h
        Fix for Bug #37408 - Compressed MyISAM files should not require/use mmap()
        
        declare 'myisam_mmap_size' and 'myisam_mmap_used' variables and the mutex
        THR_LOCK_myisam_mmap
     @ myisam/mi_packrec.c
        Fix for Bug #37408 - Compressed MyISAM files should not require/use mmap()
        
        add 'myisam_mmap_size' option which limits the memory available to mmap of 
        myisam files
     @ myisam/mi_static.c
        Fix for Bug #37408 - Compressed MyISAM files should not require/use mmap()
        
        declare 'myisam_mmap_size' and 'myisam_mmap_used' variables and the mutex
        THR_LOCK_myisam_mmap
     @ myisam/myisamdef.h
        Fix for Bug #37408 - Compressed MyISAM files should not require/use mmap()
        
        move MMAP_EXTRA_MARGIN to myisam.h so that it can be used in mysqld.cc
     @ mysql-test/r/variables.result
        Fix for Bug #37408 - Compressed MyISAM files should not require/use mmap()
        
        Testcase for BUG#37408 to test the myisam_mmap_size option
     @ mysql-test/t/variables.test
        Fix for Bug #37408 - Compressed MyISAM files should not require/use mmap()
        
        Testcase for BUG#37408 to test the myisam_mmap_size option
     @ mysys/my_thr_init.c
        Fix for Bug #37408 - Compressed MyISAM files should not require/use mmap()
        
        intialize the lock THR_LOCK_myisam_mmap
     @ sql/mysqld.cc
        Fix for Bug #37408 - Compressed MyISAM files should not require/use mmap()
        
        add the 'myisam_mmap_size' option
     @ sql/set_var.cc
        Fix for Bug #37408 - Compressed MyISAM files should not require/use mmap()
        
        add the 'myisam_mmap_size' to the SHOW VARIABLES list
[17 Dec 2009 8:59] Bugs System
A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/94674

2824 Satya B	2009-12-17
      Fix for Bug#37408 - Compressed MyISAM files should not require/use mmap()
                        
      When compressed myisam files are opened, they are always memory mapped
      sometimes causing memory swapping problems.
      
      When we mmap the myisam compressed tables of size greater than the memory 
      available, the kswapd0 process utilization is very high consuming 30-40% of 
      the cpu. This happens only with linux kernels older than 2.6.9
      
      With newer linux kernels, we don't have this problem of high cpu consumption
      and this option may not be required.
       
      The option 'myisam_mmap_size' is added to limit the amount of memory used for
      memory mapping of myisam files. This option is not dynamic.
      
      The default value on 32 bit system is 4294967295 bytes and on 64 bit system it
      is 18446744073709547520 bytes.
      
      Note: Testcase only tests the option variable. The actual bug has be to 
      tested manually.
     @ include/my_global.h
        Fix for Bug #37408 - Compressed MyISAM files should not require/use mmap()
        
        define SIZE_T_MAX
     @ include/myisam.h
        Fix for Bug #37408 - Compressed MyISAM files should not require/use mmap()
        
        declare 'myisam_mmap_size' and 'myisam_mmap_used' variables and the mutex
        THR_LOCK_myisam_mmap
     @ myisam/mi_packrec.c
        Fix for Bug #37408 - Compressed MyISAM files should not require/use mmap()
        
        add 'myisam_mmap_size' option which limits the memory available to mmap of 
        myisam files
     @ myisam/mi_static.c
        Fix for Bug #37408 - Compressed MyISAM files should not require/use mmap()
        
        declare 'myisam_mmap_size' and 'myisam_mmap_used' variables and the mutex
        THR_LOCK_myisam_mmap
     @ myisam/myisamdef.h
        Fix for Bug #37408 - Compressed MyISAM files should not require/use mmap()
        
        move MMAP_EXTRA_MARGIN to myisam.h so that it can be used in mysqld.
     @ mysql-test/r/variables.result
        Fix for Bug #37408 - Compressed MyISAM files should not require/use mmap()
        
        Testcase for BUG#37408 to test the myisam_mmap_size option
     @ mysql-test/t/variables.test
        Fix for Bug #37408 - Compressed MyISAM files should not require/use mmap()
        
        Testcase for BUG#37408 to test the myisam_mmap_size option
     @ mysys/my_thr_init.c
        Fix for Bug #37408 - Compressed MyISAM files should not require/use mmap()
        
        intialize the lock THR_LOCK_myisam_mmap
     @ sql/mysqld.cc
        Fix for Bug #37408 - Compressed MyISAM files should not require/use mmap()
        
        add the 'myisam_mmap_size' option
     @ sql/set_var.cc
        Fix for Bug #37408 - Compressed MyISAM files should not require/use mmap()
        
        add the 'myisam_mmap_size' to the SHOW VARIABLES list
[17 Dec 2009 11:26] Bugs System
A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/94715

2824 Satya B	2009-12-17
      Fix for Bug#37408 - Compressed MyISAM files should not require/use mmap()
                        
      When compressed myisam files are opened, they are always memory mapped
      sometimes causing memory swapping problems.
      
      When we mmap the myisam compressed tables of size greater than the memory 
      available, the kswapd0 process utilization is very high consuming 30-40% of 
      the cpu. This happens only with linux kernels older than 2.6.9
      
      With newer linux kernels, we don't have this problem of high cpu consumption
      and this option may not be required.
       
      The option 'myisam_mmap_size' is added to limit the amount of memory used for
      memory mapping of myisam files. This option is not dynamic.
      
      The default value on 32 bit system is 4294967295 bytes and on 64 bit system it
      is 18446744073709547520 bytes.
      
      Note: Testcase only tests the option variable. The actual bug has be to 
      tested manually.
     @ include/my_global.h
        Fix for Bug #37408 - Compressed MyISAM files should not require/use mmap()
        
        define SIZE_T_MAX
     @ include/myisam.h
        Fix for Bug #37408 - Compressed MyISAM files should not require/use mmap()
        
        declare 'myisam_mmap_size' and 'myisam_mmap_used' variables and the mutex
        THR_LOCK_myisam_mmap
     @ myisam/mi_packrec.c
        Fix for Bug #37408 - Compressed MyISAM files should not require/use mmap()
        
        add 'myisam_mmap_size' option which limits the memory available to mmap of 
        myisam files
     @ myisam/mi_static.c
        Fix for Bug #37408 - Compressed MyISAM files should not require/use mmap()
        
        declare 'myisam_mmap_size' and 'myisam_mmap_used' variables and the mutex
        THR_LOCK_myisam_mmap
     @ myisam/myisamdef.h
        Fix for Bug #37408 - Compressed MyISAM files should not require/use mmap()
        
        move MEMMAP_EXTRA_MARGIN to myisam.h so that it can be used in mysqld.cc
     @ mysql-test/r/variables.result
        Fix for Bug #37408 - Compressed MyISAM files should not require/use mmap()
        
        Testcase for BUG#37408 to test the myisam_mmap_size option
     @ mysql-test/t/variables.test
        Fix for Bug #37408 - Compressed MyISAM files should not require/use mmap()
        
        Testcase for BUG#37408 to test the myisam_mmap_size option
     @ mysys/my_thr_init.c
        Fix for Bug #37408 - Compressed MyISAM files should not require/use mmap()
        
        intialize the lock THR_LOCK_myisam_mmap
     @ sql/mysqld.cc
        Fix for Bug #37408 - Compressed MyISAM files should not require/use mmap()
        
        add the 'myisam_mmap_size' option
     @ sql/set_var.cc
        Fix for Bug #37408 - Compressed MyISAM files should not require/use mmap()
        
        add the 'myisam_mmap_size' to the SHOW VARIABLES list
[17 Dec 2009 12:26] Satya B
patch queued to mysql-5.0-bugteam , mysql-5.1-bugteam and mysql-pe
[19 Dec 2009 8:29] Bugs System
Pushed into 6.0.14-alpha (revid:alik@sun.com-20091219082307-f3i4fn0tm8trb3c0) (version source revid:alik@sun.com-20091217193315-s3ck6ltg8m1zz6r3) (merge vers: 6.0.14-alpha) (pib:15)
[19 Dec 2009 8:33] Bugs System
Pushed into 5.5.1-m2 (revid:alik@sun.com-20091219082021-f34nq4jytwamozz0) (version source revid:alexey.kopytov@sun.com-20091217150904-ugetvos7v0bo83qe) (merge vers: 5.5.0-beta) (pib:15)
[19 Dec 2009 8:37] Bugs System
Pushed into mysql-next-mr (revid:alik@sun.com-20091219082213-nhjjgmphote4ntxj) (version source revid:alik@sun.com-20091217191654-rb9sci536v96viaz) (pib:15)
[23 Dec 2009 19:20] Joerg Bruehe
This fix breaks the build of 5.5.1-m2 on the platforms OS X 10.5 and 10.6 with this error message:

ld: duplicate symbol _THR_LOCK_myisam_mmap in ../../mysys/libmysys.a(my_thr_init.o) and libmyisam.a(mi_static.o)

This is to be expected, because part of this fix is the following change:
=== modified file 'myisam/mi_static.c'
--- myisam/mi_static.c  2009-10-27 13:11:06 +0000
+++ myisam/mi_static.c  2009-12-17 11:25:50 +0000
@@ -40,7 +40,8 @@ ulong myisam_concurrent_insert= 0;
 my_off_t myisam_max_temp_length= MAX_FILE_SIZE;
 ulong    myisam_bulk_insert_tree_size=8192*1024;
 ulong    myisam_data_pointer_size=4;
-
+ulonglong    myisam_mmap_size= SIZE_T_MAX, myisam_mmap_used= 0;
+pthread_mutex_t  THR_LOCK_myisam_mmap;

 static int always_valid(const char *filename __attribute__((unused)))
 {

This new declaration of the variable conflicts with the existing declarations in include/myisam.h and mysys/my_thr_init.c

I have filed this as a bug:
Bug #49898  	Fix for bug#37408 introduces a linker error
[24 Dec 2009 6:32] Bugs System
A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/95616

2830 Satya B	2009-12-24
      Bug#49898 - Fix for bug#37408 introduces a linker error
      
      the declaration of THR_LOCK_myisam_mmap in mi_static 
      is redundant as it accessible via the extern declaration 
      in include/myisam.h
     @ myisam/mi_static.c
        Bug#49898 - Fix for bug#37408 introduces a linker error
        
        Remove THR_LOCK_myisam_mmap declaration as it is redundant
[24 Dec 2009 7:10] Bugs System
A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/95622

2932 Alexander Nozdrin	2009-12-24
      Postfix for Bug#37408 (fix for Bug#49898).
[8 Jan 2010 17:45] Paul Dubois
Noted in 5.5.1, 6.0.14 changelogs.

When compressed MyISAM files were opened, they were always memory
mapped, sometimes causing memory-swapping problems. To deal with 
this, a new system variable, myisam_mmap_size, was added to limit the
amount of memory used for memory mapping of MyISAM files.

Setting report to NDI pending push to 5.0.x, 5.1.x, Celosia.
[8 Jan 2010 17:49] Paul Dubois
"The default value on 32 bit system is 4294967295 bytes and on 64 bit system it is 18446744073709547520 bytes."

Satya, what are the minimum and maximum values?
[11 Jan 2010 12:36] Satya B
Paul,

Minimum value is 7 bytes (MEMMAP_EXTRA_MARGIN value) and the maximum is same as the default.
[11 Jan 2010 16:36] Paul Dubois
Satya, thanks, I'll update the manual with these values.
[14 Jan 2010 8:27] Bugs System
Pushed into 5.0.90 (revid:joro@sun.com-20100114082402-05fod2h6z9x9wok8) (version source revid:satya.bn@sun.com-20091224063229-csej9q00g6y7504i) (merge vers: 5.0.89) (pib:16)
[14 Jan 2010 18:00] Paul Dubois
Noted in 5.0.90 changelog.

Setting report to NDI pending push to 5.1.x, Celosia.
[15 Jan 2010 0:25] Bugs System
Pushed into 5.5.2-m2 (revid:build@mysql.com-20100115001040-7nxzvnnng1c1qhep) (version source revid:build@mysql.com-20100115001040-7nxzvnnng1c1qhep) (merge vers: 5.5.2-m2) (pib:16)
[15 Jan 2010 2:02] Paul Dubois
Setting report to NDI pending push to 5.1.x.
[15 Jan 2010 9:01] Bugs System
Pushed into 5.1.43 (revid:joro@sun.com-20100115085139-qkh0i0fpohd9u9p5) (version source revid:satya.bn@sun.com-20091224063431-q7xgol24ufqysp8t) (merge vers: 5.1.42) (pib:16)
[15 Jan 2010 18:18] Paul Dubois
Noted in 5.1.43 changelog.
[5 Feb 2010 11:49] Bugs System
Pushed into mysql-next-mr (revid:alik@sun.com-20100204063540-9czpdmpixi3iw2yb) (version source revid:alik@sun.com-20100119163614-172adculixyu26j5) (pib:16)
[5 Feb 2010 11:56] Bugs System
Pushed into 6.0.14-alpha (revid:alik@sun.com-20100205113942-oqovjy0eoqbarn7i) (version source revid:alfranio.correia@sun.com-20100121210527-rbuheu5rnsmcakh1) (merge vers: 5.5.99-m3) (pib:16)
[5 Feb 2010 12:01] Bugs System
Pushed into 5.5.2-m2 (revid:alik@sun.com-20100203172258-1n5dsotny40yufxw) (version source revid:alexey.kopytov@sun.com-20091225105650-qletdbs0wz9sx5nc) (merge vers: 5.5.1-m2) (pib:16)
[5 Feb 2010 16:50] Paul Dubois
Setting report to Need Merge pending push to Celosia.
[12 Mar 2010 14:12] Bugs System
Pushed into 5.1.44-ndb-7.0.14 (revid:jonas@mysql.com-20100312135944-t0z8s1da2orvl66x) (version source revid:jonas@mysql.com-20100312115609-woou0te4a6s4ae9y) (merge vers: 5.1.44-ndb-7.0.14) (pib:16)
[12 Mar 2010 14:28] Bugs System
Pushed into 5.1.44-ndb-6.2.19 (revid:jonas@mysql.com-20100312134846-tuqhd9w3tv4xgl3d) (version source revid:jonas@mysql.com-20100312060623-mx6407w2vx76h3by) (merge vers: 5.1.44-ndb-6.2.19) (pib:16)
[12 Mar 2010 14:44] Bugs System
Pushed into 5.1.44-ndb-6.3.33 (revid:jonas@mysql.com-20100312135724-xcw8vw2lu3mijrhn) (version source revid:jonas@mysql.com-20100312103652-snkltsd197l7q2yg) (merge vers: 5.1.44-ndb-6.3.33) (pib:16)
[12 Mar 2010 17:49] Paul Dubois
Fixed in earlier 5.1.x, 5.5.x.