Bug #46922 crash when adding partitions and open_files_limit is reached

• Submitted: 25 Aug 2009 17:07
• Modified: 19 Dec 2009 11:09
• Reporter: Shane Bester (Platinum Quality Contributor)
• Status: Closed
• Category: MySQL Server: Partitions
• Severity: S1 (Critical)
• Version: 5.1.30, 5.1.37, 5.1.39
• OS: Any
• Assigned to: Mattias Jonsson
• CPU Architecture: Any

[25 Aug 2009 17:07] Shane Bester

Description:
5.1.39 stack trace:

mysqld.exe!mi_lock_database()[mi_locking.c:31]
mysqld.exe!ha_myisam::external_lock()[ha_myisam.cc:1884]
mysqld.exe!handler::ha_external_lock()[handler.cc:4586]
mysqld.exe!ha_partition::external_lock()[ha_partition.cc:2685]
mysqld.exe!handler::ha_external_lock()[handler.cc:4586]
mysqld.exe!unlock_external()[lock.cc:786]
mysqld.exe!mysql_unlock_tables()[lock.cc:391]
mysqld.exe!close_thread_tables()[sql_base.cc:1337]
mysqld.exe!fast_alter_partition_table()[sql_partition.cc:6383]
mysqld.exe!mysql_alter_table()[sql_table.cc:6942]
mysqld.exe!mysql_execute_command()[sql_parse.cc:2858]
mysqld.exe!mysql_parse()[sql_parse.cc:5931]
mysqld.exe!dispatch_command()[sql_parse.cc:1213]
mysqld.exe!do_command()[sql_parse.cc:854]
mysqld.exe!handle_one_connection()[sql_connect.cc:1127]
mysqld.exe!pthread_start()[my_winthread.c:87]
mysqld.exe!_callthreadstart()[thread.c:293]
mysqld.exe!_threadstart()[thread.c:277]
kernel32.dll!FlsSetValue()

How to repeat:
start a 5.1 server like this on windows:

mysqld.exe --no-defaults --console --log-warnings=2 --open-files-limit=5 --max_connections=2 --table_open_cache=1

Run this:

----
select @@global.open_files_limit; #verify it's really set low!
drop table if exists `t1`;
create table `t1` (`id` int primary key) 
engine=myisam partition by key () partitions 16;
alter table `t1` add partition partitions 15; #crash
-----

[25 Aug 2009 18:30] MySQL Verification Team

c:\dbs>c:\dbs\5.1\bin\mysql -uroot --port=3510 --prompt="mysql 5.1 >"
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 2
Server version: 5.1.38-Win X64 Source distribution

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql 5.1 >use test
Database changed
mysql 5.1 >select @@global.open_files_limit; #verify it's really set low!
+---------------------------+
| @@global.open_files_limit |
+---------------------------+
|                        14 |
+---------------------------+
1 row in set (0.00 sec)

mysql 5.1 >drop table if exists `t1`;
Query OK, 0 rows affected, 1 warning (0.00 sec)

mysql 5.1 >create table `t1` (`id` int primary key)
    -> engine=myisam partition by key () partitions 16;
Query OK, 0 rows affected (0.17 sec)

mysql 5.1 >alter table `t1` add partition partitions 15; #crash
ERROR 2013 (HY000): Lost connection to MySQL server during query
mysql 5.1 >

[25 Aug 2009 18:38] MySQL Verification Team

Thank you for the bug report. Verified as described.

[28 Sep 2009 19:04] Mattias Jonsson

This avoids the crash, but there still needs to be enhanced error handling to remove temporary files, both #TMP# partition files, and #sql-<table>.{frm,par}. The #TMP# files can be handled in cleanup_new_partitions like below, but I still need to investigate how to take proper care of the #sql-<table>.{frm,par} files.
sql/ha_partition.cc:
@@ -1280,10 +1280,58 @@
     m_file= m_added_file;
     m_added_file= NULL;

+    external_lock(ha_thd(), F_UNLCK);
     /* delete_table also needed, a bit more complex */
     close();

[30 Sep 2009 8:22] Mattias Jonsson

I will fix the crash issue in this bug, and continue to fix the cleaning up of temporary partitions in bug#47343.

[8 Oct 2009 13:26] Mattias Jonsson

This is also repeatable on Mac OS X and probably in all other OS'es too, just that the limit is per default higher.

[8 Oct 2009 13:37] Bugs System

A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/86193

3115 Mattias Jonsson	2009-10-08
      Bug#46922: crash when adding partitions and open_files_limit
      is reached
      
      Problem was bad error handling, leaving some new temporary
      partitions locked and initialized and some not yet initialized
      and locked, leading to a crash when trying to unlock the not
      yet initialized and locked partitions
      
      Solution was to unlock the already locked partitions, and not
      include any of the new temporary partitions in later unlocks
     @ mysql-test/r/partition_open_files_limit.result
        Bug#46922: crash when adding partitions and open_files_limit
        is reached
        
        New test result
     @ mysql-test/t/partition_open_files_limit-master.opt
        Bug#46922: crash when adding partitions and open_files_limit
        is reached
        
        New test opt-file for testing when open_files_limit is reached
     @ mysql-test/t/partition_open_files_limit.test
        Bug#46922: crash when adding partitions and open_files_limit
        is reached
        
        New test case testing when open_files_limit is reached
     @ sql/ha_partition.cc
        Bug#46922: crash when adding partitions and open_files_limit
        is reached
        
        When cleaning up the partitions already locked need to be unlocked,
        and not be unlocked/closed after cleaning up.

[9 Oct 2009 8:47] Mattias Jonsson

Pushed into mysql-5.1-bugteam and mysql-pe

[9 Oct 2009 10:07] Mattias Jonsson

Will update the test a bit to verify if the limit is low enough... (currently succeeds in linux 64-bit mysql-pe tree which seem to have a default lowest open_files_limit set to 16384).

[9 Oct 2009 14:12] Bugs System

A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/86379

3156 Mattias Jonsson	2009-10-09
      Bug#46922 post push update
      
      Disable the test when it will not hit the open_files_limit
     @ mysql-test/t/partition_open_files_limit.test
        Bug#46922 post push update
        
        Disable the test when it will not hit the open_files_limit

[14 Oct 2009 14:39] Bugs System

Pushed into 5.1.41 (revid:[e-mail]-20091014143611-cphb0enjlx6lpat1) (version source revid:[e-mail]-20091013071829-zc4c3go44j6re592) (merge vers: 5.1.40) (pib:13)

[15 Oct 2009 8:33] Jon Stephens

Documented bugfix in the 5.1.41 changelog as follows:

        An ALTER TABLE ... ADD PARTITION statement that caused
        open_files_limit to be exceeded led to a crash of the MySQL
        server.

Closed.

[22 Oct 2009 6:34] Bugs System

Pushed into 6.0.14-alpha (revid:[e-mail]-20091022063126-l0qzirh9xyhp0bpc) (version source revid:[e-mail]-20091019135554-s1pvptt6i750lfhv) (merge vers: 6.0.14-alpha) (pib:13)

[22 Oct 2009 7:06] Bugs System

Pushed into 5.5.0-beta (revid:[e-mail]-20091022060553-znkmxm0g0gm6ckvw) (version source revid:[e-mail]-20091019131708-bc6pv55x6287a0wc) (merge vers: 5.5.0-beta) (pib:13)

[22 Oct 2009 15:06] Jon Stephens

Also documented in the 5.5.0 and 6.0.14 changelogs. 

Closed.

[18 Dec 2009 10:29] Bugs System

Pushed into 5.1.41-ndb-7.1.0 (revid:[e-mail]-20091218102229-64tk47xonu3dv6r6) (version source revid:[e-mail]-20091218095730-26gwjidfsdw45dto) (merge vers: 5.1.41-ndb-7.1.0) (pib:15)

[18 Dec 2009 10:44] Bugs System

Pushed into 5.1.41-ndb-6.2.19 (revid:[e-mail]-20091218100224-vtzr0fahhsuhjsmt) (version source revid:[e-mail]-20091217101452-qwzyaig50w74xmye) (merge vers: 5.1.41-ndb-6.2.19) (pib:15)

[18 Dec 2009 11:00] Bugs System

Pushed into 5.1.41-ndb-6.3.31 (revid:[e-mail]-20091218100616-75d9tek96o6ob6k0) (version source revid:[e-mail]-20091217154335-290no45qdins5bwo) (merge vers: 5.1.41-ndb-6.3.31) (pib:15)

[18 Dec 2009 11:14] Bugs System

Pushed into 5.1.41-ndb-7.0.11 (revid:[e-mail]-20091218101303-ga32mrnr15jsa606) (version source revid:[e-mail]-20091218064304-ezreonykd9f4kelk) (merge vers: 5.1.41-ndb-7.0.11) (pib:15)

[19 Dec 2009 11:09] Jon Stephens

No additional changelog entries needed. Setting back to Closed state.