MySQL Bugs: #40643: mysqld crashes with SIG11 during heavy inserts after some unspecifc time

Bug #40643	mysqld crashes with SIG11 during heavy inserts after some unspecifc time
Submitted:	11 Nov 2008 16:19	Modified:	12 Dec 2008 0:10
Reporter:	Andreas Bourges	Email Updates:
Status:	Closed	Impact on me:	None
Category:	MySQL Server: MyISAM storage engine	Severity:	S2 (Serious)
Version:	5.0.51a	OS:	Linux (SLES9 SP4)
Assigned to:		CPU Architecture:	Any

Description:
we're constantly inserting a large number of rows in several MyISAM tables (in the current test 400K-600K records per minute). After a while the server crashes with signal 11:

081111 15:51:18 - mysqld got signal 11;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help diagnose
the problem, but since we have already crashed, something is definitely wrong
and this may fail.

key_buffer_size=8589934592
read_buffer_size=67104768
max_used_connections=11
max_connections=100
threads_connected=6
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_connections = 41156207 K
bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

thd=0x2b9aafbe40
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
Cannot determine thread, fp=0x44088fd8, backtrace may not be correct.
Stack range sanity check OK, backtrace follows:
0x776173746572
Stack trace seems successful - bottom reached
Please read http://dev.mysql.com/doc/mysql/en/using-stack-trace.html and follow instructions on how to resolve the stack trace
. Resolved
stack trace is much more helpful in diagnosing the problem, so please do
resolve it
Trying to get some variables.
Some pointers may be invalid and cause the dump to abort...
thd->query at 0x2a98cdd790 is invalid pointer
thd->thread_id=22781
The manual page at http://www.mysql.com/doc/en/Crashing.html contains
information that should help you find out what is causing the crash.

- the system as 16GB ram
- key_cache is 8GB
- there are only very few connections, 2 of them are doing parallel insert's
- we monitor the memory utilization, which increases from ~12% up to 30% (that's about 5GB) and when reaching 30% mysqld received SIG11 (repeatable)
- other processes use 800M-1GB of RAM, that leaves ~4GB for mysql at the moment the daemon crashes

How to repeat:
- do heavy inserting in multiple tables (~500.000 records per minute)
- after ~2 hours the server crashes (or - more likely - when memory usage hits 30%)

picture showing memory utilisation on several box affected by the problem

Attachment: memory.png (image/png, text), 133.27 KiB.

Thank you for the problem report. Please, send your my.cnf file content and the results of:

uname -a
free

Linux commands. Send also SHOW CREATE TABLE results for the table you are inserting into.

Hi,

Linux S4DE9JSAAOQ 2.6.5-7.308-smp #1 SMP Mon Dec 10 11:36:40 UTC 2007 x86_64 x86_64 x86_64 GNU/Linux

             total       used       free     shared    buffers     cached
Mem:      16401836   16161044     240792          0     123104   11782160
-/+ buffers/cache:    4255780   12146056
Swap:     15735520          8   15735512

S4DE9JSAAOQ:~>

my.cnf from one of the affected systems

Attachment: my.cnf.bug (text/plain), 4.71 KiB.

sorry - unable to reproduce the bug. We did a fresh install on all servers (using the same software revisions) and the problem did not reappear again!?

memory usage is stable at somewhere above 50%... Even more key_buffer was assigned for mysql, though.

No feedback was provided for this bug for over a month, so it is
being suspended automatically. If you are able to provide the
information that was originally requested, please do so and change
the status of the bug back to "Open".