Bug #48726 mysqld keeps crashing with SIGSEGV with myisam_use_mmap enabled
Submitted: 12 Nov 2009 12:50 Modified: 29 Nov 2011 14:59
Reporter: Dino Tsoumakis Email Updates:
Status: Closed Impact on me:
None 
Category:MySQL Server: General Severity:S1 (Critical)
Version:5.1.40 5.1.43 5.1.48, 5.1.50, 5.1.52-bzr, 5.6.1 OS:Linux (SUSE Linux Enterprise Server 10 (x86_64))
Assigned to:
Tags: myisam_use_mmap, regression
Triage: Triaged: D1 (Critical)

[12 Nov 2009 12:50] Dino Tsoumakis
Description:
Since upgrade of MySQL Server from 5.1.33 to 5.1.40 we get server crashes after running for a while:

/var/lib/mysql/mysqld.log:
-----><-----
091111 11:04:35 - mysqld got signal 11 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help diagnose
the problem, but since we have already crashed, something is definitely wrong
and this may fail.

key_buffer_size=1073741824
read_buffer_size=2097152
max_used_connections=91
max_threads=200
threads_connected=92
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 3098610 K
bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

thd: 0x2aaaefc39020
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0x416bd1a8 thread_stack 0x30000
/usr/sbin/mysqld(my_print_stacktrace+0x3a)[0x82303a]
/usr/sbin/mysqld(handle_segfault+0x1fb)[0x5b9e3b]
/lib64/libpthread.so.0[0x2b3984beec00]
/usr/sbin/mysqld(heap_scan+0x43)[0x710e33]
/usr/sbin/mysqld(_ZN7ha_heap8rnd_nextEPh+0x2d)[0x70e9ed]
/usr/sbin/mysqld(_Z13rr_sequentialP14st_read_record+0x15)[0x67ec65]
/usr/sbin/mysqld[0x61c09c]
/usr/sbin/mysqld(_Z10sub_selectP4JOINP13st_join_tableb+0xe5)[0x61af55]
/usr/sbin/mysqld[0x61ab59]
/usr/sbin/mysqld(_ZN4JOIN4execEv+0x9bb)[0x60a78b]
/usr/sbin/mysqld(_Z12mysql_selectP3THDPPP4ItemP10TABLE_LISTjR4ListIS1_ES2_jP8st_orderSB_S2_SB_yP13select_resultP18st_select_lex_unitP13st_select_lex+0x15b)[0>>>x60b6cb]
/usr/sbin/mysqld(_Z13handle_selectP3THDP6st_lexP13select_resultm+0x15f)[0x60735f]
/usr/sbin/mysqld[0x5c9619]
/usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x1ca)[0x5c4e8a]
/usr/sbin/mysqld(_Z11mysql_parseP3THDPKcjPS2_+0x171)[0x5cac01]
/usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcj+0x3f5)[0x5c3875]
/usr/sbin/mysqld(_Z10do_commandP3THD+0xd0)[0x5c3320]
/usr/sbin/mysqld(handle_one_connection+0x126)[0x5c21a6]
/lib64/libpthread.so.0[0x2b3984be7143]
/lib64/libc.so.6(__clone+0x6d)[0x2b39852638cd]
Trying to get some variables.
Some pointers may be invalid and cause the dump to abort...
thd->query at 0x14db030 = SHOW TABLE STATUS
thd->thread_id=206
thd->killed=NOT_KILLED
The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you find out what is causing the crash.
091111 11:04:35 mysqld_safe Number of processes running now: 0
091111 11:04:35 mysqld_safe mysqld restarted
091111 11:04:36 [Note] Plugin 'FEDERATED' is disabled.
091111 11:04:36 [Note] Plugin 'InnoDB' is disabled.
091111 11:04:36 [Note] Event Scheduler: Loaded 0 events
091111 11:04:36 [Note] /usr/sbin/mysqld: ready for connections.
Version: '5.1.40-log'  socket: '/var/lib/mysql/mysql.sock'  port: 3306  MySQL Community Server (GPL)
091111 11:04:36 [Note] Event Scheduler: scheduler thread started with id 1
-----><-----

cu21:/ # uname -a
Linux vm21 2.6.16.60-0.21-smp #1 SMP Tue May 6 12:41:02 UTC 2008 x86_64 x86_64 x86_64 GNU/Linux

cu21:/ # free
             total       used       free     shared    buffers     cached
Mem:       3867724    2041248    1826476          0      53576    1117600
-/+ buffers/cache:     870072    2997652
Swap:      2104472      76772    2027700

cu21:/ # cat /etc/SuSE-release
SUSE Linux Enterprise Server 10 (x86_64)
VERSION = 10
PATCHLEVEL = 2

We installed MySQL 5.1.39 on another machine (also SUSE Linux Enterprise 10), but had similar problems.
We use the Linux AMD64 / Intel EM64T generic RPM installation packages.

All Tables are MyISAM, 2 Merge tables and a view partitioned tables.

After starting the server, it keeps running for a few hours and suddenly segfaults.

How to repeat:
Install the MySQL server Linux AMD64 / Intel EM64T generic RPMs on SUSE Linux Enterprise Server 10 (x86_64) Patchlevel 2 with the my.cf from here, install some GBytes MyISAM tables and keep it running for some hours?
[12 Nov 2009 12:51] Dino Tsoumakis
A gdb backtrace of such a crash shows:
-----><-----
Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread 1092770112 (LWP 8384)]
0x00002b01bee87da7 in free () from /lib64/libc.so.6
(gdb) bt
#0  0x00002b01bee87da7 in free () from /lib64/libc.so.6
#1  0x000000000081344e in my_no_flags_free (ptr=0x2aaaefc00000) at my_malloc.c:59
#2  0x00000000007f695e in mi_close (info=0x2aaaeffa1480) at mi_close.c:60
#3  0x00000000006025a4 in closefrm (table=0x2aaaf1d49b40, free_share=true) at table.cc:1971
#4  0x00000000005f569b in intern_close_table (table=0x2aaaf1d49b40) at sql_base.cc:781
#5  0x00000000005f56b9 in free_cache_entry (table=0x2aaaf1d49b40) at sql_base.cc:803
#6  0x0000000000819e3a in my_hash_delete (hash=0xc6c4a0, record=0x2aaaf1d49b40 "�����*") at hash.c:546
#7  0x00000000005f5d91 in close_open_tables (thd=0x2aaaf42f1cb0) at sql_base.cc:1203
#8  0x00000000005f5e98 in close_thread_tables (thd=0x2aaaf42f1cb0) at sql_base.cc:1350
#9  0x00000000006ab537 in get_all_tables (thd=0x2aaaf42f1cb0, tables=0x144cda0, cond=0x0) at sql_show.cc:3488
#10 0x00000000006b22f8 in get_schema_tables_result (join=0x12b3780, executed_place=PROCESSED_BY_JOIN_EXEC) at sql_show.cc:6134
#11 0x000000000060b3cd in JOIN::exec (this=0x12b3780) at sql_select.cc:1821
#12 0x000000000060b6cb in mysql_select (thd=0x2aaaf42f1cb0, rref_pointer_array=0x2aaaf42f3cb0, tables=0x144cda0, wild_num=0, fields=@0x12b3780, conds=0x0,
    og_num=0, order=0x0, group=0x0, having=0x0, proc_param=0x0, select_options=2684635648, result=0x123d3d8, unit=0x2aaaf42f36b8, select_lex=0x2aaaf42f3ae0)
    at sql_select.cc:2394
#13 0x000000000060735f in handle_select (thd=0x2aaaf42f1cb0, lex=0x2aaaf42f3618, result=0x123d3d8, setup_tables_done_option=0) at sql_select.cc:256
#14 0x00000000005c9619 in execute_sqlcom_select (thd=0x2aaaf42f1cb0, all_tables=0x144cda0) at sql_parse.cc:5043
#15 0x00000000005c4e8a in mysql_execute_command (thd=0x2aaaf42f1cb0) at sql_parse.cc:2238
#16 0x00000000005cac01 in mysql_parse (thd=0x2aaaf42f1cb0, inBuf=0x144b7a0 "SHOW TABLE STATUS", length=17, found_semicolon=0x412246b0) at sql_parse.cc:5963
#17 0x00000000005c3875 in dispatch_command (command=COM_QUERY, thd=0x2aaaf42f1cb0, packet=0x2aaaf054fad1 "SHOW TABLE STATUS", packet_length=17)
        at sql_parse.cc:1224
#18 0x00000000005c3320 in do_command (thd=0x2aaaf42f1cb0) at sql_parse.cc:865
#19 0x00000000005c21a6 in handle_one_connection (arg=0x2aaaefc00000) at sql_connect.cc:1127
#20 0x00002b01be85f143 in start_thread () from /lib64/libpthread.so.0
#21 0x00002b01beedb8cd in clone () from /lib64/libc.so.6
#22 0x0000000000000000 in ?? ()
(gdb)

(gdb) frame 1
#1  0x000000000081344e in my_no_flags_free (ptr=0x2aaaefc00000) at my_malloc.c:59
59          free(ptr);
Current language:  auto; currently c

(gdb) x 0x2aaaefc00000
0x2aaaefc00000: Cannot access memory at address 0x2aaaefc00000
-----><-----

The part of the memory map (/proc/<pid>/maps) at that time show:
-----><-----
...
2aaaefa07000-2aaaefa5b000 rw-s 00000000 08:02 481286                     /var/lib/mysql/sitecu/ExecutionJobs.MYD
2aaaefa5b000-2aaaefa6c000 rw-s 00000000 08:02 93911                      /var/lib/mysql/sitecu/ConfigurationVariables.MYD
2aaaefc01000-2aaaefd00000 rw-p 2aaaefc01000 00:00 0
2aaaefd35000-2aaaefd39000 rw-s 00000000 08:02 335086                     /var/lib/mysql/TMNL/BC_Voice_MO_MT.MYD
2aaaefd39000-2aaaefd78000 rw-s 00000000 08:02 335242                     /var/lib/mysql/TMNL/FTP_dl_ps.MYD
...
-----><-----
[12 Nov 2009 12:53] Dino Tsoumakis
/etc/my.cnf

Attachment: my.cnf (application/octet-stream, text), 2.26 KiB.

[12 Nov 2009 13:49] Valerii Kravchuk
Thank you for the problem report.

Please, try to execute SHOW TABLE STATUS on the same database just after server restart or under lowest possible load. I am interested in these results or in confirmation of fact that crash happens even without load. Try also to get SHOW GLOBAL STATUS results before trying SHOW TABLE STATUS.
[12 Nov 2009 14:32] Dino Tsoumakis
It crashes on various statements after some time, not only SHOW TABLE STATUS.
I executed lots of SHOW TABLE STATUS statements and it didn't crash.

091111 10:59:03 - mysqld got signal 11 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help diagnose
the problem, but since we have already crashed, something is definitely wrong
and this may fail.

key_buffer_size=1073741824
read_buffer_size=2097152
max_used_connections=46
max_threads=200
threads_connected=47
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 3098610 K
bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

thd: 0x1728b00
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0x411301a8 thread_stack 0x30000
/usr/sbin/mysqld(my_print_stacktrace+0x3a)[0x82303a]
/usr/sbin/mysqld(handle_segfault+0x1fb)[0x5b9e3b]
/lib64/libpthread.so.0[0x2b3d70826c00]
/lib64/libc.so.6(memcpy+0x52)[0x2b3d70e500b2]
[0xf29a99]
Trying to get some variables.
Some pointers may be invalid and cause the dump to abort...
thd->query at 0x1a36de0 = SELECT /* MEPR */ * FROM MeasurementTargets WHERE Target = 'vfuk_hsdpa'
thd->thread_id=83
thd->killed=NOT_KILLED
The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you find out what is causing the crash.
091111 10:59:04 mysqld_safe Number of processes running now: 0
091111 10:59:04 mysqld_safe mysqld restarted
091111 10:59:04 [Note] Plugin 'FEDERATED' is disabled.
091111 10:59:04 [Note] Plugin 'InnoDB' is disabled.
091111 10:59:04 [Note] Event Scheduler: Loaded 0 events
091111 10:59:04 [Note] /usr/sbin/mysqld: ready for connections.
Version: '5.1.40-log'  socket: '/var/lib/mysql/mysql.sock'  port: 3306  MySQL Community Server (GPL)
...
...
091111 11:07:38 - mysqld got signal 11 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help diagnose
the problem, but since we have already crashed, something is definitely wrong
and this may fail.

key_buffer_size=1073741824
read_buffer_size=2097152
max_used_connections=60
max_threads=200
threads_connected=60
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 3098610 K
bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

thd: 0x1370670
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0x4140f1a8 thread_stack 0x30000
/usr/sbin/mysqld(my_print_stacktrace+0x3a)[0x82303a]
/usr/sbin/mysqld(handle_segfault+0x1fb)[0x5b9e3b]
/lib64/libpthread.so.0[0x2b2d14e35c00]
/usr/sbin/mysqld(heap_scan+0x43)[0x710e33]
/usr/sbin/mysqld(_ZN7ha_heap8rnd_nextEPh+0x2d)[0x70e9ed]
/usr/sbin/mysqld(_Z13rr_sequentialP14st_read_record+0x15)[0x67ec65]
/usr/sbin/mysqld[0x61c09c]
/usr/sbin/mysqld(_Z10sub_selectP4JOINP13st_join_tableb+0xe5)[0x61af55]
/usr/sbin/mysqld[0x61ab59]
/usr/sbin/mysqld(_ZN4JOIN4execEv+0x9bb)[0x60a78b]
/usr/sbin/mysqld(_Z12mysql_selectP3THDPPP4ItemP10TABLE_LISTjR4ListIS1_ES2_jP8st_orderSB_S2_SB_yP13select_resultP18st_select_lex_unitP13st_select_lex+0x15b)[0>>>x60b6cb]
/usr/sbin/mysqld(_Z13handle_selectP3THDP6st_lexP13select_resultm+0x15f)[0x60735f]
/usr/sbin/mysqld[0x5c9619]
/usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x1ca)[0x5c4e8a]
/usr/sbin/mysqld(_Z11mysql_parseP3THDPKcjPS2_+0x171)[0x5cac01]
/usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcj+0x3f5)[0x5c3875]
/usr/sbin/mysqld(_Z10do_commandP3THD+0xd0)[0x5c3320]
/usr/sbin/mysqld(handle_one_connection+0x126)[0x5c21a6]
/lib64/libpthread.so.0[0x2b2d14e2e143]
/lib64/libc.so.6(__clone+0x6d)[0x2b2d154aa8cd]
Trying to get some variables.
Some pointers may be invalid and cause the dump to abort...
thd->query at 0x2aaaf0c68fb0 is an invalid pointer
thd->thread_id=173
thd->killed=NOT_KILLED
The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you find out what is causing the crash.
091111 11:07:39 mysqld_safe Number of processes running now: 0
091111 11:07:39 mysqld_safe mysqld restarted
091111 11:07:39 [Note] Plugin 'FEDERATED' is disabled.
091111 11:07:39 [Note] Plugin 'InnoDB' is disabled.
091111 11:07:39 [Note] Event Scheduler: Loaded 0 events
091111 11:07:39 [Note] /usr/sbin/mysqld: ready for connections.
Version: '5.1.40-log'  socket: '/var/lib/mysql/mysql.sock'  port: 3306  MySQL Community Server (GPL)
091111 11:07:39 [Note] Event Scheduler: scheduler thread started with id 1
...

BTW: We downgraded one system to 5.1.33 for the moment and it is running for more than 2 hour without any crash using the same my.cnf.
[18 Nov 2009 19:13] Valerii Kravchuk
If you still do not have crashes in 5.1.33, please, try to repeat with a newer version, 5.1.41, and inform about the results.
[19 Dec 2009 0:00] Bugs System
No feedback was provided for this bug for over a month, so it is
being suspended automatically. If you are able to provide the
information that was originally requested, please do so and change
the status of the bug back to "Open".
[2 Feb 2010 13:36] Dino Tsoumakis
After installing the latest 5.1.43 generic 64bit rpms on the same SLES 10 64bit system, it is crashing again sporadically, but quite often (nearly every 10 minutes). Same situation on other systems, too.

Again:
We have 5.1.33 with same config running for more than 3 month without any crash!

Please investigate!

See output of mysqld.log with debuginfos attached.
[2 Feb 2010 13:37] Dino Tsoumakis
mysqld.log with debuginfos

Attachment: mysqld.log (application/octet-stream, text), 0 bytes.

[2 Feb 2010 13:38] Dino Tsoumakis
mysqld.log with debuginfos

Attachment: mysqld.log (text/x-log), 6.81 KiB.

[17 Jun 2010 11:49] Valerii Kravchuk
Please, check if the same problem still happens with a newer version, 5.1.48.
[17 Jun 2010 12:44] Dino Tsoumakis
What a coincidence!
Due to other bugs we ran into, we tested the latest 5.1.48 release version today.
The server kept crashing every few minutes.
Then I deactivated the myisam_use_mmap option in the config and ... it's working fine! No crash since hours!

I used a 64bit SUSE Linux Enterprise Server 10 with 8GB RAM.
So for me the solution seems to be to disable myisam_use_mmap.
[30 Jun 2010 0:32] Miguel Solorzano
Are you able to provide a repeatable test case crash enabling myisam_use_mmap?. Thanks in advance.
[30 Jul 2010 23:00] Bugs System
No feedback was provided for this bug for over a month, so it is
being suspended automatically. If you are able to provide the
information that was originally requested, please do so and change
the status of the bug back to "Open".
[29 Sep 2010 8:22] Shane Bester
this reliably crashes my 5.1.50 on fedora core 13. Can somebody test else this please?  import with --force option in client

Attachment: tables.sql (application/octet-stream, text), 38.70 KiB.

[29 Sep 2010 9:22] Valerii Kravchuk
Verified as described in the last comment from Shane with current mysql-5.1 from bzr on Ubuntu 10.04:

openxs@ubuntu:/home2/openxs/dbs/5.1$ bin/mysqld_safe --no-defaults --myisam_use_mmap=1 --socket=/tmp/vk.sock --port=9999 &
[1] 1720
openxs@ubuntu:/home2/openxs/dbs/5.1$ 100929 12:16:46 mysqld_safe Logging to '/home2/openxs/dbs/5.1/var/ubuntu.err'.
100929 12:16:46 mysqld_safe Starting mysqld daemon with databases from /home2/openxs/dbs/5.1/var

openxs@ubuntu:/home2/openxs/dbs/5.1$ bin/mysql --no-defaults -uroot --force --socket=/tmp/vk.sock test < ~/Desktop/tables.sql 
ERROR 1048 (23000) at line 90: Column 'a3' cannot be null
ERROR 1048 (23000) at line 108: Column 'col16' cannot be null
ERROR 1048 (23000) at line 132: Column 'col15' cannot be null
ERROR 1048 (23000) at line 167: Column 'col21' cannot be null
ERROR 1048 (23000) at line 188: Column 'col1' cannot be null
ERROR 1048 (23000) at line 210: Column 'a2' cannot be null
ERROR 1048 (23000) at line 301: Column 'a1' cannot be null
ERROR 1048 (23000) at line 438: Column 'col19' cannot be null
ERROR 1048 (23000) at line 523: Column 'a2' cannot be null
ERROR 1048 (23000) at line 569: Column 'col1' cannot be null
ERROR 1048 (23000) at line 627: Column 'col4' cannot be null
ERROR 1048 (23000) at line 666: Column 'a6' cannot be null
ERROR 1048 (23000) at line 683: Column 'col16' cannot be null
ERROR 1048 (23000) at line 761: Column 'a8' cannot be null
ERROR 1048 (23000) at line 877: Column 'col15' cannot be null
ERROR 1048 (23000) at line 903: Column 'col4' cannot be null
ERROR 1048 (23000) at line 944: Column 'a9' cannot be null
ERROR 1048 (23000) at line 1076: Column 'col21' cannot be null
ERROR 1048 (23000) at line 1095: Column 'a5' cannot be null
ERROR 1048 (23000) at line 1198: Column 'a8' cannot be null
ERROR 1048 (23000) at line 1211: Column 'a2' cannot be null
ERROR 1048 (23000) at line 1463: Column 'col19' cannot be null
ERROR 1048 (23000) at line 1470: Column 'col21' cannot be null
ERROR 1048 (23000) at line 1513: Column 'a1' cannot be null
ERROR 1048 (23000) at line 1689: Column 'col1' cannot be null
ERROR 1048 (23000) at line 1715: Column 'a6' cannot be null
ERROR 1048 (23000) at line 1751: Column 'a1' cannot be null
ERROR 1048 (23000) at line 1768: Column 'a7' cannot be null
ERROR 1048 (23000) at line 1828: Column 'col22' cannot be null
ERROR 1048 (23000) at line 1945: Column 'a5' cannot be null
ERROR 1048 (23000) at line 1991: Column 'a9' cannot be null
ERROR 1048 (23000) at line 2150: Column 'a6' cannot be null
ERROR 1048 (23000) at line 2171: Column 'a7' cannot be null
ERROR 2013 (HY000) at line 2212: Lost connection to MySQL server during query
ERROR 2006 (HY000) at line 2220: MySQL server has gone away
ERROR 2006 (HY000) at line 2222: MySQL server has gone away
ERROR 2006 (HY000) at line 2224: MySQL server has gone away
openxs@ubuntu:/home2/openxs/dbs/5.1$ 100929 12:19:03 mysqld_safe Number of processes running now: 0
100929 12:19:03 mysqld_safe mysqld restarted
^C
openxs@ubuntu:/home2/openxs/dbs/5.1$ tail -50 var/ubuntu.err

thd: 0xa521798
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0xb43b838c thread_stack 0x30000
/home2/openxs/dbs/5.1/libexec/mysqld(my_print_stacktrace+0x26)[0x85fb220]
/home2/openxs/dbs/5.1/libexec/mysqld(handle_segfault+0x2b9)[0x827abda]
[0xda1400]
/home2/openxs/dbs/5.1/libexec/mysqld[0x858dd54]
/home2/openxs/dbs/5.1/libexec/mysqld(_mi_delete_dynamic_record+0x30)[0x858cd11]
/home2/openxs/dbs/5.1/libexec/mysqld(mi_delete+0x3e2)[0x8598446]
/home2/openxs/dbs/5.1/libexec/mysqld(_ZN9ha_myisam10delete_rowEPKh+0x31)[0x85ada75]
/home2/openxs/dbs/5.1/libexec/mysqld(_ZN7handler13ha_delete_rowEPKh+0x33)[0x83c4d5f]
/home2/openxs/dbs/5.1/libexec/mysqld(_Z12write_recordP3THDP8st_tableP12st_copy_info+0x938)[0x8322b94]
/home2/openxs/dbs/5.1/libexec/mysqld(_Z12mysql_insertP3THDP10TABLE_LISTR4ListI4ItemERS3_IS5_ES6_S6_15enum_duplicatesb+0xd55)[0x8320d1a]
/home2/openxs/dbs/5.1/libexec/mysqld(_Z21mysql_execute_commandP3THD+0x3670)[0x828fccf]
/home2/openxs/dbs/5.1/libexec/mysqld(_Z11mysql_parseP3THDPcjPPKc+0x265)[0x82989dc]
/home2/openxs/dbs/5.1/libexec/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcj+0xbee)[0x828a8e1]
/home2/openxs/dbs/5.1/libexec/mysqld(_Z10do_commandP3THD+0x26c)[0x82899e9]
/home2/openxs/dbs/5.1/libexec/mysqld(handle_one_connection+0x159)[0x8287b8a]
/lib/tls/i686/cmov/libpthread.so.0(+0x596e)[0xb3396e]
/lib/tls/i686/cmov/libc.so.6(clone+0x5e)[0x786a4e]
Trying to get some variables.
Some pointers may be invalid and cause the dump to abort...
thd->query at 0xa54a3c8 = replace  into `test`.`t5` set
`col10` = 'wxn',
`col7` = 1,
`col8` = 9.22337203685E+18,
`col9` = '2011-02-28',
`a5` = 4294967296
thd->thread_id=1
thd->killed=NOT_KILLED
The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you find out what is causing the crash.
100929 12:19:03 mysqld_safe Number of processes running now: 0
100929 12:19:03 mysqld_safe mysqld restarted
100929 12:19:03 [Note] Plugin 'FEDERATED' is disabled.
InnoDB: Log scan progressed past the checkpoint lsn 0 84682849
100929 12:19:03  InnoDB: Database was not shut down normally!
InnoDB: Starting crash recovery.
InnoDB: Reading tablespace information from the .ibd files...
InnoDB: Restoring possible half-written data pages from the doublewrite
InnoDB: buffer...
InnoDB: Doing recovery: scanned up to log sequence number 0 84683323
InnoDB: Last MySQL binlog file position 0 429, file name ./ubuntu-bin.000002
100929 12:19:03  InnoDB: Started; log sequence number 0 84683323
100929 12:19:04 [Note] Event Scheduler: Loaded 0 events
100929 12:19:04 [Note] /home2/openxs/dbs/5.1/libexec/mysqld: ready for connections.
Version: '5.1.52-valgrind-max-debug'  socket: '/tmp/vk.sock'  port: 9999  Source distribution
[30 Sep 2010 5:07] Shane Bester
bug #56793 is a duplicate of this
[11 Oct 2010 15:32] Shane Bester
just a note to folks who run into this bug.
you can avoid it by:

1) myisam_use_mmap=0
2) try make a huge table_open_cache so that tables aren't swapped in/out the table cache.
[30 May 2011 11:09] Shane Bester
bug #61351 is a duplicate of this.
[18 Jun 2011 16:35] Chris Wagner
If this has been a known bug for so long why hasn't there been a fix yet or atleast a severe caveat in the manual!!  Or even removed from the product.
[27 Oct 2011 8:27] Nils Goroll
I've seen two crashes with presumably the same root cause.

Server version: 5.1.52-log Source distribution

Linux hostname 2.6.32-131.17.1.el6.x86_64 #1 SMP Thu Sep 29 10:24:25 EDT 2011 x86_64 x86_64 x86_64 GNU/Linux

have removed myisam_use_mmap for the time being.

111026 20:05:28 - mysqld got signal 11 ;
...
stack_bottom = 0x7fe49e624d98 thread_stack 0x40000
/usr/libexec/mysqld(my_print_stacktrace+0x29) [0x847ba9]
/usr/libexec/mysqld(handle_segfault+0x415) [0x5baae5]
/lib64/libpthread.so.0() [0x3f24e0f490]
/lib64/libc.so.6(memcpy+0xa0) [0x3f24a87f40]
/usr/libexec/mysqld(mi_mmap_pread+0x9c) [0x80e20c]
/usr/libexec/mysqld(_mi_read_dynamic_record+0x22b) [0x80d32b]
/usr/libexec/mysqld(mi_rkey+0x2c3) [0x801963]
/usr/libexec/mysqld(ha_myisam::index_read_map(unsigned char*, unsigned char
const*, unsigned long, ha_r
key_function)+0x54) [0x7f9214]
/usr/libexec/mysqld(handler::read_range_first(st_key_range const*, st_key_range
const*, bool, bool)+0xb
e) [0x6907ae]
/usr/libexec/mysqld(handler::read_multi_range_first(st_key_multi_range**,
st_key_multi_range*, unsigned
 int, bool, st_handler_buffer*)+0xd1) [0x6908e1]
/usr/libexec/mysqld(QUICK_RANGE_SELECT::get_next()+0x156) [0x678bf6]
/usr/libexec/mysqld() [0x68b9f1]
/usr/libexec/mysqld(sub_select(JOIN*, st_join_table*, bool)+0x69) [0x61cd49]
/usr/libexec/mysqld() [0x61d61e]
/usr/libexec/mysqld(JOIN::exec()+0xbac) [0x62f2ac]
/usr/libexec/mysqld(mysql_select(THD*, Item***, TABLE_LIST*, unsigned int,
List<Item>&, Item*, unsigned
 int, st_order*, st_order*, Item*, st_order*, unsigned long long,
select_result*, st_select_lex_unit*,
st_select_lex*)+0x14a) [0x62b15a]
/usr/libexec/mysqld(handle_select(THD*, st_lex*, select_result*, unsigned
long)+0x174) [0x6308b4]
/usr/libexec/mysqld() [0x5c4bba]
/usr/libexec/mysqld(mysql_execute_command(THD*)+0x4be) [0x5c6cae]
/usr/libexec/mysqld(mysql_parse(THD*, char*, unsigned int, char const**)+0x2d3)
[0x5cc143]
/usr/libexec/mysqld(dispatch_command(enum_server_command, THD*, char*, unsigned
int)+0x4f9) [0x5ccad9]
/usr/libexec/mysqld(do_command(THD*)+0xea) [0x5cddfa]
/usr/libexec/mysqld(handle_one_connection+0x23d) [0x5c117d]
/lib64/libpthread.so.0() [0x3f24e077e1]
/lib64/libc.so.6(clone+0x6d) [0x3f24ae577d]

111026 22:23:42 - mysqld got signal 11 ;
...
stack_bottom = 0x7fb6476ebd98 thread_stack 0x40000
/usr/libexec/mysqld(my_print_stacktrace+0x29) [0x847ba9]
/usr/libexec/mysqld(handle_segfault+0x415) [0x5baae5]
/lib64/libpthread.so.0() [0x3f24e0f490]
/lib64/libc.so.6(memcpy+0x1e) [0x3f24a87ebe]
/usr/libexec/mysqld(mi_mmap_pread+0x9c) [0x80e20c]
/usr/libexec/mysqld(_mi_read_dynamic_record+0x22b) [0x80d32b]
/usr/libexec/mysqld(mi_rkey+0x2c3) [0x801963]
/usr/libexec/mysqld(ha_myisam::index_read_map(unsigned char*, unsigned char
const*, unsigned long, ha_r
key_function)+0x54) [0x7f9214]
/usr/libexec/mysqld() [0x618345]
/usr/libexec/mysqld(sub_select(JOIN*, st_join_table*, bool)+0x69) [0x61cd49]
/usr/libexec/mysqld() [0x612332]
/usr/libexec/mysqld(sub_select(JOIN*, st_join_table*, bool)+0x76) [0x61cd56]
/usr/libexec/mysqld() [0x61d61e]
/usr/libexec/mysqld(JOIN::exec()+0xbac) [0x62f2ac]
/usr/libexec/mysqld(mysql_select(THD*, Item***, TABLE_LIST*, unsigned int,
List<Item>&, Item*, unsigned
 int, st_order*, st_order*, Item*, st_order*, unsigned long long,
select_result*, st_select_lex_unit*,
st_select_lex*)+0x14a) [0x62b15a]
/usr/libexec/mysqld(handle_select(THD*, st_lex*, select_result*, unsigned
long)+0x174) [0x6308b4]
/usr/libexec/mysqld() [0x5c4bba]
/usr/libexec/mysqld(mysql_execute_command(THD*)+0x4be) [0x5c6cae]
/usr/libexec/mysqld(mysql_parse(THD*, char*, unsigned int, char const**)+0x2d3)
[0x5cc143]
/usr/libexec/mysqld(dispatch_command(enum_server_command, THD*, char*, unsigned
int)+0x4f9) [0x5ccad9]
/usr/libexec/mysqld(do_command(THD*)+0xea) [0x5cddfa]
/usr/libexec/mysqld(handle_one_connection+0x23d) [0x5c117d]
/lib64/libpthread.so.0() [0x3f24e077e1]
/lib64/libc.so.6(clone+0x6d) [0x3f24ae577d]
Trying to get some variables.
Some pointers may be invalid and cause the dump to abort...
thd->query at 0x7fb70c0babb0 is an invalid pointer
thd->thread_id=181175
thd->killed=NOT_KILLED
[10 Nov 2011 8:54] Shane Bester
Just a quick note to say this bug still hits latest mysql-trunk (5.6.4-m7) on linux.   The settings were --table_open_cache=10 --myisam-use-mmap=1
and 1000 random myisam tables being inserted into, and a mysqldump dumping them during the process.

(gdb) bt
#0  in pthread_kill () from /lib64/libpthread.so.0
#1  in my_write_core (sig=11) at ./mysys/stacktrace.c:423
#2  in handle_segfault (sig=11) at ./sql/mysqld.cc:2676
#3  <signal handler called>
#4  in memcpy () from /lib64/libc.so.6
#5  in mi_mmap_pwrite at ./storage/myisam/mi_dynrec.c:237
#6  in _mi_write_part_record at ./storage/myisam/mi_dynrec.c:766
#7  in update_dynamic_record  at ./storage/myisam/mi_dynrec.c:939
#8  in _mi_update_blob_record  at ./storage/myisam/mi_dynrec.c:318
#9  in mi_update at ./storage/myisam/mi_update.c:155
#10 in ha_myisam::update_row at ./storage/myisam/ha_myisam.cc:1555
#11 in handler::ha_update_row at ./sql/handler.cc:6080
#12 in write_record at ./sql/sql_insert.cc:1639
#13 in mysql_insert at ./sql/sql_insert.cc:948
#14 in mysql_execute_command at ./sql/sql_parse.cc:3136
#15 in mysql_parse at ./sql/sql_parse.cc:5859
#16 in dispatch_command at ./sql/sql_parse.cc:1193
#17 in do_command at ./sql/sql_parse.cc:916
#18 in do_handle_one_connection at ./sql/sql_connect.cc:790
#19 in handle_one_connection at ./sql/sql_connect.cc:706
#20 in start_thread from /lib64/libpthread.so.0
#21 in clone from /lib64/libc.so.6
[29 Nov 2011 14:59] Paul Dubois
Noted in 5.1.61, 5.5.20, 5.6.5 changelogs.

Enabling myisam_use_mmap could cause the server to crash.
[11 Dec 2011 19:26] Harald Reindl
> Noted in 5.1.61, 5.5.20, 5.6.5 changelogs.
> Enabling myisam_use_mmap could cause the server to crash.

well NOTED is not fixed :-)

i tried this option long ago with 5.1.x
with 5.5.18 i tried it again and after two days a server with
4000 tables crashed again, and yes it was myisam_use_mmap

will this ever work or do we still buy better performance with random crashes?
[12 Dec 2011 1:44] Shane Bester
Harald, 5.5.20 will contain a fix.
This means it is expected that 5.5.18 and 5.5.19 releases are broken still.
[7 Aug 2013 8:31] Shane Bester
May affect myisampack'd tables.

From the ChangeLog:
    revno: 3560.6.66
    committer: Nirbhay Choubey <nirbhay.choubey@oracle.com>
    branch nick: B11756764-5.1
    timestamp: Tue 2011-11-29 17:59:35 +0530
    message:
      Bug#11756764 48726: MYSQLD KEEPS CRASHING WITH SIGSEGV
                          WITH MYISAM_USE_MMAP ENABLED
      
      MySQL server can crash due to segmentation fault when
      started with myisam_use_mmap.
      
      The reason behind this being, while making a request to
      unmap (munmap) the previously mapped memory (mmap), the
      size passed was 7 bytes larger than the size requested at
      the time of mapping. This can eventually unmap the adjacent
      memory mapped block, belonging to some other memory-map pool.
      Hence the subsequent call to mmap can map a region which was
      still a valid memory mapped area.
      
      Fixed by removing the extra 7-byte margin which was erroneously
      added to the size, used for unmappping.
[6 Mar 2014 8:24] Abhijit Buchake
my.cnf file

Attachment: bug_my.cnf (application/octet-stream, text), 2.73 KiB.

[6 Mar 2014 8:24] Abhijit Buchake
I got this error today on a production server even though myisam_use_mmap is OFF. Following are the details.

OS: SunOS 5.10 Generic
MySQL Version: 5.1.56-log
RAM: 72G
CPUs: Intel Xeon CPUs 2 nos.

+------------------+----------------------+
| Variable_name    | Value                |
+------------------+----------------------+
| myisam_mmap_size | 18446744073709551615 |
| myisam_use_mmap  | OFF                  |
+------------------+----------------------+

Any suggestions?
[6 Mar 2014 9:32] Shane Bester
Abhijit, I suggest you plan an upgrade to the latest version, 5.1.73, 5.5.36, or 5.6.16.  This bug has been solved.  If you still experience problems on current version, please open a new bug report.