Bug #70185 | Mysql 5.6.13 - InnoDB: Unable to allocate memory of size 18446744073709546280 | ||
---|---|---|---|
Submitted: | 29 Aug 2013 18:20 | Modified: | 23 Jan 2014 0:36 |
Reporter: | Josiah Webb | Email Updates: | |
Status: | Can't repeat | Impact on me: | |
Category: | MySQL Server: InnoDB storage engine | Severity: | S1 (Critical) |
Version: | 5.6.1x | OS: | Linux (3.4.43-43.43.amzn1.x86_64) |
Assigned to: | CPU Architecture: | Any | |
Tags: | innodb memory buffer |
[29 Aug 2013 18:20]
Josiah Webb
[29 Aug 2013 19:16]
MySQL Verification Team
Looks similar to http://bugs.mysql.com/bug.php?id=70073 please see [19 Aug 5:17] Shane Bester comment. Thanks.
[29 Aug 2013 20:25]
Josiah Webb
Thanks for getting back to me on this... I mentioned that i had already tried that suggested fix: "However, just adding 'innodb_stats_persistent=0' to my.cnf alone does *not* fix the problem" Mysql will not start unless i do 1 of the 2 things i mentioned ('innodb_force_recovery=4' or 'innodb_change_buffering=none' in my.cnf). innodb_stats_persistent=0 does nothing for the error.
[29 Aug 2013 21:02]
Josiah Webb
Also, to be clear.. the amount of memory that innodb is attempting to allocate is 18446744073709551615 bytes (17 billion GB's), so it's unlikely that it's allocating for innodb stats. It's is likely related to the innodb insert buffer as i described in the "suggested fix" section above.
[29 Aug 2013 21:39]
MySQL Verification Team
But you did the 'delete'?: DELETE FROM mysql.innodb_index_stats; DELETE FROM mysql.innodb_table_stats; That stopped the assertion for that reporter. I had hope was the same case because the assertion printed by InnoDB happens in the same place.
[30 Aug 2013 3:23]
Josiah Webb
So the only way mysqld will start is for me to add either 'innodb_force_recovery=4' or 'innodb_change_buffering=none' to my.cnf. I cannot use 'innodb_force_recovery=4' as it prevents me from doing any DELETE statements on innodb tables. So, i added 'innodb_change_buffering=none' to my.cnf and mysqld started, then i ran the delete statements from http://bugs.mysql.com/bug.php?id=70073. mysql> DELETE FROM mysql.innodb_index_stats; Query OK, 1112767 rows affected (8.95 sec) mysql> DELETE FROM mysql.innodb_table_stats; Query OK, 91044 rows affected (1.99 sec) Stopped mysqld properly and without any issues. Removed 'innodb_change_buffering=none' from my.cnf and started mysqld back up. Same exact error... same exact place: 2013-08-30 03:14:20 4532 [ERROR] InnoDB: InnoDB: Unable to allocate memory of size 18446744073709540984. 2013-08-30 03:14:20 7f31e4b96700 InnoDB: Assertion failure in thread 139852267480832 in file ha_innodb.cc line 16865
[30 Aug 2013 5:23]
MySQL Verification Team
We need to know the stack trace. Can you either resolve the stack trace into function names, or run mysqld in gdb ? gdb /usr/sbin/mysqld set arg --defaults-file=/path/to/my.cnf --console --gdb run ........ bt bt full
[31 Aug 2013 1:55]
Josiah Webb
backtrace
Attachment: backtrace.txt (text/plain), 6.14 KiB.
[31 Aug 2013 1:56]
Josiah Webb
backtrace full
Attachment: backtrace-full.txt (text/plain), 17.72 KiB.
[31 Aug 2013 1:56]
Josiah Webb
Thanks for the response... backtraces attached to this bug!
[31 Aug 2013 7:42]
MySQL Verification Team
(gdb) bt #0 in raise () from /lib64/libc.so.6 #1 in abort () from /lib64/libc.so.6 #2 in ib_logf at ./storage/innobase/handler/ha_innodb.cc:16865 #3 in mem_heap_create_block at ./storage/innobase/mem/mem0mem.cc:360 #4 in mem_heap_add_block at ./storage/innobase/mem/mem0mem.cc:452 #5 in mem_heap_alloc at ./storage/innobase/include/mem0mem.ic:186 #6 in mem_heap_dup at ./storage/innobase/mem/mem0mem.cc:125 #7 in trx_undo_rec_copy at ./storage/innobase/include/trx0rec.ic:111 #8 in trx_undo_get_undo_rec_low at ./storage/innobase/trx/trx0rec.cc:1429 #9 in trx_undo_get_undo_rec at ./storage/innobase/trx/trx0rec.cc:1461 #10 in trx_undo_prev_version_build at ./storage/innobase/trx/trx0rec.cc:1537 #11 in row_vers_old_has_index_entry at ./storage/innobase/row/row0vers.cc:421 #12 in row_purge_poss_sec at ./storage/innobase/row/row0purge.cc:252 #13 in btr_cur_search_to_nth_level at ./storage/innobase/btr/btr0cur.cc:661 #14 in btr_pcur_open_low at ./storage/innobase/include/btr0pcur.ic:440 #15 in row_search_index_entry at ./storage/innobase/row/row0row.cc:815 #16 in row_purge_remove_sec_if_poss_leaf at ./storage/innobase/row/row0purge.cc:421 #17 in row_purge_remove_sec_if_poss at ./storage/innobase/row/row0purge.cc:482 #18 in row_purge_upd_exist_or_extern_func at ./storage/innobase/row/row0purge.cc:581 #19 in row_purge_record_func at ./storage/innobase/row/row0purge.cc:818 #20 in row_purge at ./storage/innobase/row/row0purge.cc:862 #21 in row_purge_step at ./storage/innobase/row/row0purge.cc:942 #22 in que_thr_step at ./storage/innobase/que/que0que.cc:1105 #23 in que_run_threads_low at ./storage/innobase/que/que0que.cc:1167 #24 in que_run_threads at ./storage/innobase/que/que0que.cc:1208 #25 in trx_purge at ./storage/innobase/trx/trx0purge.cc:1246 #26 in srv_do_purge at ./storage/innobase/srv/srv0srv.cc:2582 #27 in srv_purge_coordinator_thread at ./storage/innobase/srv/srv0srv.cc:2759 #28 in start_thread from /lib64/libpthread.so.0 #29 in clone from /lib64/libc.so.6 The problem is "len" is absurd here, or undo_rec points to garbage? trx_undo_rec_copy( /*==============*/ const trx_undo_rec_t* undo_rec, /*!< in: undo log record */ mem_heap_t* heap) /*!< in: heap where copied */ { ulint len; len = mach_read_from_2(undo_rec) - ut_align_offset(undo_rec, UNIV_PAGE_SIZE); ut_ad(len < UNIV_PAGE_SIZE); return((trx_undo_rec_t*) mem_heap_dup(heap, undo_rec, len)); Similar crashes once seen in internal Bug #14235384 - SEGV IN UNDO PROCESSING I guess mysqld-debug or debug build will assert earlier.
[1 Sep 2013 9:47]
MySQL Verification Team
Kindly send us the complete mysql error log so we can check previous startups, upgrades, etc.
[4 Sep 2013 15:01]
Josiah Webb
We toasted this EC2 instance over the weekend (AWS cluster compute instance are expensive to have sitting around doing nothing). I am trying from scratch again (this was already done 3 times prior w/ the same erroneous results) and will report if i experience the same error. The instance i am creating now is using a more "vanilla" AWS AMI (Bashton CentOS-6.4). I'll let you know in the next day or 2 and will be able to provide complete logs for you if it recurs.
[23 Jan 2014 0:36]
MySQL Verification Team
Setting "can't repeat" here until we see this again.