Bug #9487 myisampack segmentation violation and bus error

Description:
I have various segmentation faults and bus-errors when I run 
myisampack Ver 1.22 for sun-solaris2.8 on sparc
from the
mysqld  Ver 4.1.5-gamma-standard for sun-solaris2.8 on sparc 
(Official MySQL-standard binary) distribution.

I do not think that this problem is related with bug 6110
http://bugs.mysql.com/bug.php?id=6110

I have sets of identical tables that I want to compress and then MERGE JOIN and myisampack crashes for some of them.

All my tables have no indexes.
I will give more details for the smallest in size tables that I have found the 
problem for, but I have several more examples

I have  tables residue_data_<1-8> and for all tables 
residue_data_1-3 and 5-8 compress OK (myisamchk -r gives no 
errors neither before or after the compression, the tables seem to work 
properly and can also be uncompressed) but myisampack residue_data_4 
(3618984 rows - 840 MB) gives segmentation violation just after the statistics
are calculated and the compression starts. 

(In a similar fashion for another set of tables: atom_data_<1-32> I get segmentation violation in 5 cases and bus errors in 2).

When I copy the residue_data_4 files on my local Windows machine where I run myisampack 
myisampack.exe Ver 1.22 for Win95/Win98 on i32, 
from
mysqld  Ver 4.1.2-alpha-debug for Win95/Win98 on i32 (Source distribution)

the compression works OK and when I move the compressed files back to solaris, everything works OK myisamchk -r etc. so I have a workaround for myself.

Uncompressing this file also works OK and compressing it back again gives the  
same core dump.
The stack from the core-dump (using dbx that I have available on solaris is)

=>[1] _malloc_unlocked(0x10000, 0x5452202020202020, 0xffffffff7e4b0f80, 0x10000, 0x1004488f0, 0x0), at 0xffffffff7e349350
  [2] malloc(0x10000, 0x4, 0x2, 0x0, 0x0, 0x0), at 0xffffffff7e3491c8
  [3] my_malloc(0x10000, 0x20, 0x13ff7, 0x13ff8, 0x414, 0xfffffffffffffffc), at 0x100038b98
  [4] init_io_cache(0x100237d90, 0x3fff, 0x20, 0x0, 0x0, 0x4000), at 0x100037330
  [5] mi_extra(0x321a1464, 0x3, 0xfff8, 0xffffff, 0x1, 0x0), at 0x100013a78
  [6] 0x10000f0cc(0xffffffff7ffff780, 0xffffffff7fffeb80, 0xfc00, 0x49500, 0x1, 0x5), at 0x10000f0cb
  [7] 0x10000e120(0x226, 0x10023cbd0, 0x4, 0x100286c50, 0xffffffff7ffff780, 0xffffffff7fffeb80), at 0x10000e11f
  [8] 0x10000b88c(0xffffffff7ffff780, 0x0, 0x1, 0x230000, 0x0, 0x67), at 0x10000b88b
  [9] main(0xffffffff7ffff778, 0x0, 0xffffffff7ffff8a0, 0x100231938, 0x100000000, 0x0), at 0x10000ad40

I have put the compressed "not-causing core dump" table
residue_data_1 and the problematic one residue_data_4
on the web together with the create table script
and the core file at
http://www.ebi.ac.uk/~dimitris/myisampack_core.tar.gz
I know that the structure of this table seems to be very complex
but I have also similar problems with the simpler atom_data table 

Thank you very much for you time.

How to repeat:
Download http://www.ebi.ac.uk/~dimitris/myisampack_core.tar.gz (500 MB)
gzip -d myisampack_core.tar.gz
tar xvf myisampack_core.tar
Use mysqld  Ver 4.1.5-gamma-standard for sun-solaris2.8 on sparc 
(Official MySQL-standard binary) distribution. and
myisampack Ver 1.22 for sun-solaris2.8 on sparc
run
myisampack residue_data_4

I could not repeat this on Linux. I have to try this remotely on a sun now.

Could not reproduce with mysql-debug-4.1.5-gamma-sun-solaris2.8-sparc. Now trying 64-bit version.

OK, I can repeat with mysql-debug-4.1.5-gamma-sun-solaris2.8-sparc-64bit. This might be a memory overrun. I need to use a version with safe malloc. So I try with a current version first.

A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/internals/24059

This was in fact two bugs. One was improper casting for 64-bit systems. The other one should have occured every now and then. Perhaps fortunate arrangements of malloced segments allowed for a buffer overflow.

Noted in 4.1.12 changelog.