MySQL Bugs: #48469: when innodb tablespace is configured too small, crash and corruption!

Bug #48469	when innodb tablespace is configured too small, crash and corruption!
Submitted:	2 Nov 2009 12:48	Modified:	18 Jun 2010 23:02
Reporter:	Shane Bester (Platinum Quality Contributor)	Email Updates:
Status:	Closed	Impact on me:	None
Category:	MySQL Server: InnoDB storage engine	Severity:	S1 (Critical)
Version:	5.0.85, 5.1.40, 5.4.3	OS:	Any
Assigned to:	Satya B	CPU Architecture:	Any

Description:
Version: '5.1.40-enterprise-gpl-advanced'  socket: ''  port: 3306  MySQL Enterprise Server - Advanced Edition (GPL)
091102 14:44:59InnoDB: Error: trying to access a stray pointer 0000000004343FF8
InnoDB: buf pool start is at 0000000004344000, end at 0000000004B44000
InnoDB: Probable reason is database corruption or memory
InnoDB: corruption. If this happens in an InnoDB database recovery, see
InnoDB: http://dev.mysql.com/doc/refman/5.1/en/forcing-recovery.html
InnoDB: how to force recovery.
091102 14:44:59  InnoDB: Assertion failure in thread 3448 in file G:\mysql-5.1.40-winbuild\mysql-advanced-gpl-5.1.40-build\storage\innobase\include\buf0buf.ic line 264
InnoDB: We intentionally generate a memory trap.

mysqld.exe!xdes_get_offset()[fsp0fsp.c:789]
mysqld.exe!fseg_free_step()[fsp0fsp.c:3238]
mysqld.exe!btr_free_but_not_root()[btr0btr.c:774]
mysqld.exe!dict_drop_index_tree()[dict0crea.c:676]
mysqld.exe!row_undo_ins_remove_clust_rec()[row0uins.c:61]
mysqld.exe!row_undo()[row0undo.c:282]
mysqld.exe!row_undo_step()[row0undo.c:331]
mysqld.exe!que_thr_step()[que0que.c:1254]
mysqld.exe!que_run_threads_low()[que0que.c:1322]
mysqld.exe!que_run_threads()[que0que.c:1357]
mysqld.exe!trx_general_rollback_for_mysql()[trx0roll.c:84]
mysqld.exe!row_create_index_for_mysql()[row0mysql.c:2075]
mysqld.exe!create_index()[ha_innodb.cc:5348]
mysqld.exe!ha_innobase::create()[ha_innodb.cc:5562]
mysqld.exe!ha_create_table()[handler.cc:3606]
mysqld.exe!rea_create_table()[unireg.cc:418]
mysqld.exe!mysql_create_table_no_lock()[sql_table.cc:3931]
mysqld.exe!mysql_create_table()[sql_table.cc:4027]
mysqld.exe!mysql_execute_command()[sql_parse.cc:2726]
mysqld.exe!mysql_parse()[sql_parse.cc:5967]
mysqld.exe!dispatch_command()[sql_parse.cc:1226]
mysqld.exe!do_command()[sql_parse.cc:865]
mysqld.exe!handle_one_connection()[sql_connect.cc:1127]
mysqld.exe!pthread_start()[my_winthread.c:85]
mysqld.exe!_callthreadstart()[thread.c:295]
mysqld.exe!_threadstart()[thread.c:275]

How to repeat:
on a clean installation, start mysqld like this:

mysqld --innodb_data_file_path=ibdata1:10M

then import the two tables attached.

Suggested fix:
Either fix error handling properly for this bug and bug #43665 , or make the limit a 'soft limit' and error out when it's safe to do so.

import this into a clean tablespace

Attachment: bug48469_testcase.sql (application/unknown, text), 5.42 KiB.

5.0, 5.1, 5.4 all crash with different assertions/failure messages.

Verified just as described with recent 5.1.41-debug from bzr on Mac OS X. Assertion I've got is on different line, but still:

091102 16:00:33 [Warning] Got signal: 18  error: 0
091102 16:01:16InnoDB: Error: trying to access a stray pointer 0x2503ff8
InnoDB: buf pool start is at 0x2504000, end at 0x2d04000
InnoDB: Probable reason is database corruption or memory
InnoDB: corruption. If this happens in an InnoDB database recovery, see
InnoDB: http://dev.mysql.com/doc/refman/5.1/en/forcing-recovery.html
InnoDB: how to force recovery.
091102 16:01:16  InnoDB: Assertion failure in thread 2956922880 in file ../../storage/innobase/include/buf0buf.ic line 225
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: http://dev.mysql.com/doc/refman/5.1/en/forcing-recovery.html
InnoDB: about forcing recovery.
091102 16:01:16 - mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help diagnose
the problem, but since we have already crashed, something is definitely wrong
and this may fail.

key_buffer_size=8384512
read_buffer_size=131072
max_used_connections=1
max_threads=151
threads_connected=1
It is possible that mysqld could use up to 
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 337715 K
bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

thd: 0x0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0x0 thread_stack 0x30000
0   mysqld                              0x0058919a my_print_stacktrace + 45
1   mysqld                              0x000fcfc6 handle_segfault + 882
2   libSystem.B.dylib                   0x940472bb _sigtramp + 43
3   ???                                 0xffffffff 0x0 + 4294967295
4   mysqld                              0x001074a3 main + 1581
5   mysqld                              0x00001926 start + 54
The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you find out what is causing the crash.
091102 16:01:16 mysqld_safe mysqld restarted

In btr_create(), fseg_create() could fail, and return NULL, however, its status not checked, page_no 256 continue to be returned. This page causing the rollback failure when we free it. Verifying this finding.

Stack:

fsp_try_extend_data_file <=== Failed 
fsp_reserve_free_extents
fseg_create_general
fseg_create
btr_create  <=== Did not check return status for fseg_create, return 256
dict_create_index_tree_step
dict_create_index_step

==== Some Debug Info =======================================
2816            success = fsp_try_extend_data_file(&n_pages_added, space,
(gdb) step
fsp_try_extend_data_file (actual_increase=0x45300130, space=0,
    header=0x2aaaaafb4026 "", mtr=0x45300340) at fsp/fsp0fsp.c:1101
1101            *actual_increase = 0;
(gdb) n
1103            if (space == 0 && !srv_auto_extend_last_data_file) {
(gdb)
1105                    return(FALSE);
(gdb) n
1186    }
(gdb)
fsp_reserve_free_extents (n_reserved=0x453001f8, space=0, n_ext=2,
    alloc_type=1000000, mtr=0x45300340) at fsp/fsp0fsp.c:2818
2818            if (success && n_pages_added > 0) {
(gdb) n
2823            return(FALSE);
(gdb) n
2824    }
(gdb)
fseg_create_general (space=0, page=256, byte_offset=74,
    has_done_reservation=0, mtr=0x45300340) at fsp/fsp0fsp.c:2066
2066                    if (!success) {
(gdb) n
2067                            return(NULL);
(gdb) n
2134    }
(gdb) n
fseg_create (space=0, page=256, byte_offset=74, mtr=0x45300340)
    at fsp/fsp0fsp.c:2155
2155    }
(gdb) n
btr_create (type=0, space=0, index_id={high = 0, low = 91}, comp=1,
    mtr=0x45300340) at btr/btr0btr.c:722
722             page = page_create(frame, mtr, comp);
(gdb) n
723             buf_block_align(page)->check_index_page_at_flush = TRUE;
(gdb) p page
$17 = (ib_page_t *) 0x2aaaab3b8000 ""
(gdb) p page_no
$18 = 256
================================================================

This page 256 is causing deallocation problem. Using debugger skip 
this page, all other free fine.

#0  0x0000003c1a230265 in raise () from /lib64/libc.so.6
#1  0x0000003c1a231d10 in abort () from /lib64/libc.so.6
#2  0x00000000008291b8 in buf_block_align (ptr=0x2aaaaafb3ff9 "")
    at ../../storage/innobase/include/buf0buf.ic:225
#3  0x00000000008292d5 in buf_frame_get_page_no (ptr=0x2aaaaafb3ff9 "")
    at ../../storage/innobase/include/buf0buf.ic:280
#4  0x0000000000846201 in xdes_get_offset (descr=0x2aaaaafb3ff9 "")
    at fsp/fsp0fsp.c:786
#5  0x0000000000849d9f in fseg_free_step (header=0x2aaaab3b804a "",
    mtr=0x452ffde0) at fsp/fsp0fsp.c:3236
#6  0x00000000008a9c1c in btr_free_but_not_root (space=0, root_page_no=256) <==
    at btr/btr0btr.c:772
#7  0x00000000008b0945 in dict_drop_index_tree (rec=0x2aaaaafe14f6 "",
    mtr=0x45300330) at dict/dict0crea.c:668
#8  0x00000000008b75a1 in row_undo_ins_remove_clust_rec (node=0x2aaaaaddd8b8)
    at row/row0uins.c:59
#9  0x00000000008b7b39 in row_undo_ins (node=0x2aaaaaddd8b8)
    at row/row0uins.c:305
#10 0x000000000088a951 in row_undo (node=0x2aaaaaddd8b8, thr=0x2aaaaad92db0)
    at row/row0undo.c:280
#11 0x000000000088aa0d in row_undo_step (thr=0x2aaaaad92db0)

==============

A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/92028

3214 Satya B	2009-11-30
      Applying InnoDB snapshot 5.1-ss6242, part 3. Fixes BUG#48469
      
      1. BUG#4869 - when innodb tablespace is configured too small,
                    crash and corruption!
      
      Detailed revision comments:
      
      r6187 | jyang | 2009-11-18 05:27:30 +0200 (Wed, 18 Nov 2009) | 9 lines
      branches/5.1: Fix bug #48469 "when innodb tablespace is
      configured too small, crash and corruption!". Function
      btr_create() did not check the return status of fseg_create(),
      and continue the index creation even there is no sufficient
      space.
      
      rb://205 Approved by Marko
      
      r6200 | vasil | 2009-11-19 12:14:23 +0200 (Thu, 19 Nov 2009) | 4 lines
      branches/5.1:
      
      White space fixup - indent under the opening (
      
      r6203 | jyang | 2009-11-19 15:12:22 +0200 (Thu, 19 Nov 2009) | 8 lines
      branches/5.1: Use btr_free_root() instead of fseg_free() for
      the fix of bug #48469, because fseg_free() is not defined
      in the zip branch. And we could save one mini-trasaction started
      by fseg_free().
      
      Approved by Marko.

A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/92091

3230 Satya B	2009-11-30
      Applying InnoDB Plugin 1.0.6 snapshot, Fixes BUG#48469 
      
      applied revisions: r6201, r6202, r6207, r6208, r6210
      
      r6202 - port fix for BUG#48469 to plugin
      
      Detailed revision comments:
      
      r6201 | marko | 2009-11-19 14:09:11 +0200 (Thu, 19 Nov 2009) | 2 lines
      branches/zip: ha_innobase::add_index(): Clarify the comment
      on orphaned tables when creating a primary key.
      r6202 | jyang | 2009-11-19 15:01:00 +0200 (Thu, 19 Nov 2009) | 8 lines
      branches/zip: Function fseg_free() is no longer defined
      in branches/zip. To port fix for bug #48469 to zip,
      we can use btr_free_root() which frees the page,
      and also does not require mini-transaction.
      
      Approved by Marko.
      
      
      r6207 | vasil | 2009-11-20 10:19:14 +0200 (Fri, 20 Nov 2009) | 54 lines
      branches/zip: Merge r6198:6206 from branches/5.1:
      
      (r6203 was skipped as it is already in branches/zip)
      
        ------------------------------------------------------------------------
        r6200 | vasil | 2009-11-19 12:14:23 +0200 (Thu, 19 Nov 2009) | 4 lines
        Changed paths:
           M /branches/5.1/btr/btr0btr.c
        
        branches/5.1:
        
        White space fixup - indent under the opening (
        
        ------------------------------------------------------------------------
        r6203 | jyang | 2009-11-19 15:12:22 +0200 (Thu, 19 Nov 2009) | 8 lines
        Changed paths:
           M /branches/5.1/btr/btr0btr.c
        
        branches/5.1: Use btr_free_root() instead of fseg_free() for
        the fix of bug #48469, because fseg_free() is not defined
        in the zip branch. And we could save one mini-trasaction started
        by fseg_free().
        
        Approved by Marko.
        
        
        ------------------------------------------------------------------------
        r6205 | jyang | 2009-11-20 07:55:48 +0200 (Fri, 20 Nov 2009) | 11 lines
        Changed paths:
           M /branches/5.1/handler/ha_innodb.cc
        
        branches/5.1: Add a special case to handle the Duplicated Key error
        and return DB_ERROR instead. This is to avoid a possible SIGSEGV
        by mysql error handling re-entering the storage layer for dup key
        info without proper table handle.
        This is to prevent a server crash when error situation in bug
        #45961 "DDL on partitioned innodb tables leaves data dictionary
        in an inconsistent state" happens.
        
        rb://157 approved by Sunny Bains.
        
        
        ------------------------------------------------------------------------
        r6206 | jyang | 2009-11-20 09:38:43 +0200 (Fri, 20 Nov 2009) | 5 lines
        Changed paths:
           M /branches/5.1/handler/ha_innodb.cc
        
        branches/5.1: Fix a minor code formating issue for 
        the parenthesis iplacement of the if condition in
        rename_table().
        
        
        ------------------------------------------------------------------------
      
      r6208 | vasil | 2009-11-20 10:49:24 +0200 (Fri, 20 Nov 2009) | 4 lines
      branches/zip:
      
      Add ChangeLog entry for c6207.
      
      r6210 | vasil | 2009-11-20 23:39:48 +0200 (Fri, 20 Nov 2009) | 3 lines
      branches/zip:
      
      Whitespace fixup.

patch queued to 5.1-bugteam storage/innobase and for the plugin storage/innodb_
plugin.

NULL merged to 6.0 and will be merged to 5.5.*

Pushed into 5.1.42 (revid:joro@sun.com-20091202080033-mndu4sxwx19lz2zs) (version source revid:satya.bn@sun.com-20091130134226-8n07m0minjyhey9y) (merge vers: 5.1.42) (pib:13)

Noted in 5.1.42 changelog.

If the InnoDB tablespace was configured with too small a value, the
server could crash and corrupt the tablespace.

Setting report to NDI pending push to 5.5.x.

Pushed into 6.0.14-alpha (revid:alik@sun.com-20091216083311-xorsasf5kopjxshf) (version source revid:alik@sun.com-20091214191830-wznm8245ku8xo702) (merge vers: 6.0.14-alpha) (pib:14)

Pushed into 5.5.0-beta (revid:alik@sun.com-20091216082430-s0gtzibcgkv4pqul) (version source revid:satya.bn@sun.com-20091202140050-nh3ebk6s3bziv8cb) (merge vers: 5.5.0-beta) (pib:14)

Pushed into mysql-next-mr (revid:alik@sun.com-20091216083231-rp8ecpnvkkbhtb27) (version source revid:alik@sun.com-20091212203859-fx4rx5uab47wwuzd) (merge vers: 5.6.0-beta) (pib:14)

Noted in 5.5.1, 6.0.14 changelogs.

this crash could also be seen if you did many CREATE TEMPORARY TABLE and the tmpdir filled up while using innodb_file_per_table

Pushed into 5.1.44-ndb-7.0.14 (revid:jonas@mysql.com-20100312135944-t0z8s1da2orvl66x) (version source revid:jonas@mysql.com-20100312115609-woou0te4a6s4ae9y) (merge vers: 5.1.44-ndb-7.0.14) (pib:16)

Pushed into 5.1.44-ndb-6.2.19 (revid:jonas@mysql.com-20100312134846-tuqhd9w3tv4xgl3d) (version source revid:jonas@mysql.com-20100312060623-mx6407w2vx76h3by) (merge vers: 5.1.44-ndb-6.2.19) (pib:16)

Pushed into 5.1.44-ndb-6.3.33 (revid:jonas@mysql.com-20100312135724-xcw8vw2lu3mijrhn) (version source revid:jonas@mysql.com-20100312103652-snkltsd197l7q2yg) (merge vers: 5.1.44-ndb-6.3.33) (pib:16)

Pushed into 5.1.47 (revid:joro@sun.com-20100505145753-ivlt4hclbrjy8eye) (version source revid:vasil.dimov@oracle.com-20100331130613-8ja7n0vh36a80457) (merge vers: 5.1.46) (pib:16)

Push resulted from incorporation of InnoDB tree. No changes pertinent to this bug.
Re-closing.

Pushed into mysql-next-mr (revid:alik@sun.com-20100524190136-egaq7e8zgkwb9aqi) (version source revid:vasil.dimov@oracle.com-20100331130613-8ja7n0vh36a80457) (pib:16)

Pushed into 6.0.14-alpha (revid:alik@sun.com-20100524190941-nuudpx60if25wsvx) (version source revid:vasil.dimov@oracle.com-20100331130613-8ja7n0vh36a80457) (merge vers: 5.1.46) (pib:16)

Pushed into 5.5.5-m3 (revid:alik@sun.com-20100524185725-c8k5q7v60i5nix3t) (version source revid:vasil.dimov@oracle.com-20100331130613-8ja7n0vh36a80457) (merge vers: 5.1.46) (pib:16)

Push resulted from incorporation of InnoDB tree. No changes pertinent to this bug.
Re-closing.

Pushed into 5.5.5-m3 (revid:alik@sun.com-20100615080459-smuswd9ooeywcxuc) (version source revid:mmakela@bk-internal.mysql.com-20100415070122-1nxji8ym4mao13ao) (merge vers: 5.1.47) (pib:16)

Pushed into mysql-next-mr (revid:alik@sun.com-20100615080558-cw01bzdqr1bdmmec) (version source revid:mmakela@bk-internal.mysql.com-20100415070122-1nxji8ym4mao13ao) (pib:16)

Pushed into 5.1.47-ndb-7.0.16 (revid:martin.skold@mysql.com-20100617114014-bva0dy24yyd67697) (version source revid:vasil.dimov@oracle.com-20100331130613-8ja7n0vh36a80457) (merge vers: 5.1.46) (pib:16)

Pushed into 5.1.47-ndb-6.2.19 (revid:martin.skold@mysql.com-20100617115448-idrbic6gbki37h1c) (version source revid:vasil.dimov@oracle.com-20100331130613-8ja7n0vh36a80457) (merge vers: 5.1.46) (pib:16)

Pushed into 5.1.47-ndb-6.3.35 (revid:martin.skold@mysql.com-20100617114611-61aqbb52j752y116) (version source revid:vasil.dimov@oracle.com-20100331130613-8ja7n0vh36a80457) (merge vers: 5.1.46) (pib:16)