Bug #47645 Segmentation fault when out of memory during handlerton initialization

• Submitted: 25 Sep 2009 9:24
• Modified: 13 Nov 2009 16:03
• Reporter: Mats Kindahl
• Status: Closed
• Category: MySQL Server: Storage Engine API
• Severity: S2 (Serious)
• Version: 5.1.40
• OS: Any
• Assigned to: Mats Kindahl
• CPU Architecture: Any

[25 Sep 2009 9:24] Mats Kindahl

Description:
When allocation fails with an out of memory condition in ha_initialize_handlerton, the init method for will be called with NULL. If the handlertons init method expect a non-NULL value, it will cause a crash when trying to de-reference the pointer.

How to repeat:
Not easily.

Suggested fix:
Do not call the init() function if memory allocation failed.

[25 Sep 2009 9:35] Bugs System

A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/84596

3110 Mats Kindahl	2009-09-25
      Bug #47645: Segmentation fault when out of memory during handlerton initialization
      
      There is a missing check for memory allocation failure when allocating
      memory for the handlerton structure. If the handlerton init function
      tries to de-reference the pointer, it will cause a segmentation fault
      and crash the server.
      
      This patch fixes the problem by not calling the init function if memory
      allocation failed, and instead reports the error to the caller.
     @ sql/handler.cc
        Add a check if memory allocation succeeded before calling the init
        function. If it failed, it is not necessary to free the memory,
        but the plugin->data is set to NULL to ensure that it can be checked
        for failure.

[25 Sep 2009 9:48] Bugs System

A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/84603

3110 Mats Kindahl	2009-09-25
      Bug #47645: Segmentation fault when out of memory during handlerton initialization
      
      There is a missing check for memory allocation failure when allocating
      memory for the handlerton structure. If the handlerton init function
      tries to de-reference the pointer, it will cause a segmentation fault
      and crash the server.
      
      This patch fixes the problem by not calling the init function if memory
      allocation failed, and instead prints an informative error message and
      reports the error to the caller.
     @ sql/handler.cc
        Add a check if memory allocation succeeded before calling the init
        function. If it failed, it is not necessary to free the memory,
        but the plugin->data is set to NULL to ensure that it can be checked
        for failure.

[1 Oct 2009 22:23] Luis Soares

This was pushed to mysql-5.1-rep+2:

http://pb2.norway.sun.com/web.py?template=push_details&push=542014

Setting to PQ.

[27 Oct 2009 9:48] Bugs System

Pushed into 6.0.14-alpha (revid:[e-mail]-20091027094604-9p7kplu1vd2cvcju) (version source revid:[e-mail]-20091026140226-uhnqejkyqx1aeilc) (merge vers: 6.0.14-alpha) (pib:13)

[30 Oct 2009 11:08] MC Brown

Internal change only; no changelog entry required.

[12 Nov 2009 8:17] Bugs System

Pushed into 5.5.0-beta (revid:[e-mail]-20091110093229-0bh5hix780cyeicl) (version source revid:[e-mail]-20091027095744-rf45u3x3q5d1f5y0) (merge vers: 5.5.0-beta) (pib:13)

[13 Nov 2009 16:03] MC Brown

Internal only; no documentation needed