MySQL Bugs: #62292: Server restarting after signal 11

Bug #62292	Server restarting after signal 11
Submitted:	29 Aug 2011 16:07	Modified:	13 Dec 2011 17:39
Reporter:	Pedro Mesquita	Email Updates:
Status:	No Feedback	Impact on me:	None
Category:	MySQL Server	Severity:	S1 (Critical)
Version:	5.1.49-3	OS:	Linux (Debian squeeze 2.6.32-5-amd64)
Assigned to:		CPU Architecture:	Any
Tags:	Server.restarting.after.signal.11

Description:
I'm geeting a signal 11 error. Server runs fine for a random period of time and then crashes and restarts with all tables marked as crashed. Already moved to another server but the problem still persists with same error.
Currently Mysqlserver runs on a DellpowerEdge 2950 with 16Gb of mem and cpu 3.2 Mhz on scsi raid10 with jfs file system. 

Aug 29 06:01:44 DBServer mysqld: 110829  6:01:44 - mysqld got signal 11 ;
Aug 29 06:01:44 DBServer mysqld: This could be because you hit a bug. It is also poABible that this binary
Aug 29 06:01:44 DBServer mysqld: or one of the libraries it was linked against is corrupt, improperly built,
Aug 29 06:01:44 DBServer mysqld: or misconfigured. This error can also be caused by malfunctioning hardware.
Aug 29 06:01:44 DBServer mysqld: We will try our best to scrape up some info that will hopefully help diagnose
Aug 29 06:01:44 DBServer mysqld: the problem, but since we have already crashed, something is definitely wrong
Aug 29 06:01:44 DBServer mysqld: and this may fail.
Aug 29 06:01:44 DBServer mysqld: 
Aug 29 06:01:44 DBServer mysqld: key_buffer_size=1572864000
Aug 29 06:01:44 DBServer mysqld: read_buffer_size=131072
Aug 29 06:01:44 DBServer mysqld: max_used_connections=1157
Aug 29 06:01:44 DBServer mysqld: max_threads=2048
Aug 29 06:01:44 DBServer mysqld: threads_connected=406
Aug 29 06:01:44 DBServer mysqld: It is poABible that mysqld could use up to 
Aug 29 06:01:44 DBServer mysqld: key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 6013536 K
Aug 29 06:01:44 DBServer mysqld: bytes of memory
Aug 29 06:01:44 DBServer mysqld: Hope that's ok; if not, decrease some variables in the equation.
Aug 29 06:01:44 DBServer mysqld: 
Aug 29 06:01:44 DBServer mysqld: thd: 0x7fd23806b040
Aug 29 06:01:44 DBServer mysqld: Attempting backtrace. You can use the following information to find out
Aug 29 06:01:44 DBServer mysqld: where mysqld died. If you see no meABages after this, something went
Aug 29 06:01:44 DBServer mysqld: terribly wrong...
Aug 29 06:01:44 DBServer mysqld: stack_bottom = 0x7fd7f3260e88 thread_stack 0x40000
Aug 29 06:01:44 DBServer mysqld: /usr/sbin/mysqld(my_print_stacktrace+0x29) [0x7fd8fb045829]
Aug 29 06:01:44 DBServer mysqld: /usr/sbin/mysqld(handle_segfault+0x404) [0x7fd8fad4fb14]
Aug 29 06:01:44 DBServer mysqld: /lib/libpthread.so.0(+0xef60) [0x7fd8fa5b1f60]
Aug 29 06:01:44 DBServer mysqld: /lib/libpthread.so.0(pthread_create+0x10d) [0x7fd8fa5a9d3d]
Aug 29 06:01:44 DBServer mysqld: /usr/sbin/mysqld(mysql_insert(THD*, TABLE_LIST*, List<Item>&, List<List<Item> >&, List<Item>&, List<Item>&, enum_duplicates, bool)+0xb30) [0x7fd8fadd95f0]
Aug 29 06:01:44 DBServer mysqld: /usr/sbin/mysqld(mysql_execute_command(THD*)+0xbc9) [0x7fd8fad624a9]
Aug 29 06:01:44 DBServer mysqld: /usr/sbin/mysqld(mysql_parse(THD*, char const*, unsigned int, char const**)+0x3fb) [0x7fd8fad6730b]
Aug 29 06:01:44 DBServer mysqld: /usr/sbin/mysqld(dispatch_command(enum_server_command, THD*, char*, unsigned int)+0xb34) [0x7fd8fad67e54]
Aug 29 06:01:44 DBServer mysqld: /usr/sbin/mysqld(do_command(THD*)+0xea) [0x7fd8fad68d3a]
Aug 29 06:01:44 DBServer mysqld: /usr/sbin/mysqld(handle_one_connection+0x235) [0x7fd8fad5aa25]
Aug 29 06:01:44 DBServer mysqld: /lib/libpthread.so.0(+0x68ba) [0x7fd8fa5a98ba]
Aug 29 06:01:44 DBServer mysqld: /lib/libc.so.6(clone+0x6d) [0x7fd8f90f202d]
Aug 29 06:01:44 DBServer mysqld: Trying to get some variables.
Aug 29 06:01:44 DBServer mysqld: Some pointers may be invalid and cause the dump to abort...
Aug 29 06:01:44 DBServer mysqld: thd->query at 0x7fd2384f4b60 is an invalid pointer
Aug 29 06:01:44 DBServer mysqld: thd->thread_id=2203880
Aug 29 06:01:44 DBServer mysqld: thd->killed=NOT_KILLED
Aug 29 06:01:44 DBServer mysqld: The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
Aug 29 06:01:44 DBServer mysqld: information that should help you find out what is causing the crash.
Aug 29 06:01:44 DBServer mysqld: pure virtual method called
Aug 29 06:01:44 DBServer mysqld: terminate called without an active exception
Aug 29 06:01:45 DBServer mysqld_safe: Number of proceABes running now: 0
Aug 29 06:01:45 DBServer mysqld_safe: mysqld restarted
Aug 29 06:01:45 DBServer mysqld: 110829  6:01:45 [Warning] '--log_slow_queries' is deprecated and will be removed in a future release. Please use ''--slow_query_log'/'--slow_query_log_file'' instead.
Aug 29 06:01:46 DBServer mysqld: InnoDB: Log scan progreABed past the checkpoint lsn 328 2039904705
Aug 29 06:01:46 DBServer mysqld: 110829  6:01:46  InnoDB: Database was not shut down normally!
Aug 29 06:01:46 DBServer mysqld: InnoDB: Starting crash recovery.
Aug 29 06:01:46 DBServer mysqld: InnoDB: Reading tablespace information from the .ibd files...
Aug 29 06:01:46 DBServer mysqld: InnoDB: Restoring poABible half-written data pages from the doublewrite
Aug 29 06:01:46 DBServer mysqld: InnoDB: buffer...
Aug 29 06:01:46 DBServer mysqld: InnoDB: Doing recovery: scanned up to log sequence number 328 2045008792
Aug 29 06:01:46 DBServer mysqld: 110829  6:01:46  InnoDB: Starting an apply batch of log records to the database...
Aug 29 06:01:59 DBServer mysqld: InnoDB: ProgreAB in percents: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 
Aug 29 06:02:00 DBServer mysqld: InnoDB: Apply batch completed

How to repeat:
It cannot be repeted at will.

my.cnf and error

Attachment: fullerrorandmy.cnf.txt (text/plain), 11.15 KiB.

I think that crash can be a result of out of memory condition and/or use of mmap. Look:

key_buffer		= 1500M
query_cache_size                = 256M
innodb_buffer_pool_size = 2G
innodb_additional_mem_pool_size = 20M

So, we have almost 3G of memory just for global buffers, then you allow a lot of concurrent connections (and 1000+ were really used):

max_connections        = 2048

max_heap_table_size = 1610612736 #currently not using
tmp_table_size = 1610612736		

and allow temporary tables to use up to 1.6G of memory before going to disk. Now, with just 10 big tables like this, you can easy end up with out of memory. On top of this add:

myisam_use_mmap = 1		

and I am not surprised you have crash and tables marked as corrupted.

Please, check if the same problem ever happens with

# myisam_use_mmap = 1		

commented out and

max_heap_table_size = 16M
tmp_table_size = 16M

No feedback was provided for this bug for over a month, so it is
being suspended automatically. If you are able to provide the
information that was originally requested, please do so and change
the status of the bug back to "Open".