Bug #36638 mysqld crashes when open file limit is passed and general query log enabled
Submitted: 9 May 2008 22:32 Modified: 15 Sep 2008 19:04
Reporter: Omer Barnir (OCA) Email Updates:
Status: Closed Impact on me:
None 
Category:MySQL Server: CSV Severity:S1 (Critical)
Version:5.1.26-glibc23, 5.1, 5.1.28bzr OS:Any
Assigned to: Sergei Glukhov CPU Architecture:Any

[9 May 2008 22:32] Omer Barnir
Description:
If the number of open files required is higher then the the number set on the system, the server will crash with a signal 11 after reporting error:24 in the master.err file. The result stack trace is recorded:
/home/qauser/src5125d_0508/sql/mysqld[0x82c57c]
/home/qauser/src5125d_0508/sql/mysqld(print_stacktrace+0x15)[0x82c5ba]
/home/qauser/src5125d_0508/sql/mysqld(handle_segfault+0x2ee)[0x681a1e]
/lib64/libpthread.so.0[0x2b3807752140]
/home/qauser/src5125d_0508/sql/mysqld(safe_mutex_lock+0x1c)[0xac4848]
/home/qauser/src5125d_0508/sql/mysqld(thr_lock+0xa0)[0xac2521]
/home/qauser/src5125d_0508/sql/mysqld(thr_multi_lock+0xae)[0xac38a1]
/home/qauser/src5125d_0508/sql/mysqld(_Z17mysql_lock_tablesP3THDPP8st_tablejjPb+0x37d)[0x6785cb]
/home/qauser/src5125d_0508/sql/mysqld(_Z11open_ltableP3THDP10TABLE_LIST13thr_lock_typej+0x22d)[0x6e1e19]
/home/qauser/src5125d_0508/sql/mysqld(_Z29open_performance_schema_tableP3THDP10TABLE_LISTP17Open_tables_state+0x82)[0x6e1f00]
/home/qauser/src5125d_0508/sql/mysqld(_ZN24Log_to_csv_event_handler11log_generalEP3THDlPKcjiS3_jS3_jP15charset_info_st+0x147)[0x74edcf]
/home/qauser/src5125d_0508/sql/mysqld(_ZN6LOGGER17general_log_writeEP3THD19enum_server_commandPKcj+0x28b)[0x74d927]
/home/qauser/src5125d_0508/sql/mysqld(_Z17general_log_writeP3THD19enum_server_commandPKcj+0x46)[0x74d9d6]
/home/qauser/src5125d_0508/sql/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcj+0xbcc)[0x69bdd6]
/home/qauser/src5125d_0508/sql/mysqld(_Z10do_commandP3THD+0x234)[0x69d1d0]
/home/qauser/src5125d_0508/sql/mysqld(handle_one_connection+0x137)[0x68aa07]
/lib64/libpthread.so.0[0x2b380774b193]
/lib64/libc.so.6(__clone+0x6d)[0x2b3807ed845d]

How to repeat:
generate a load on the system that requires a number of open files that is higher then the one set by the system (such as created using the QA system test suite (however this is not specifically related to the scenario run by the test suite). 

Suggested fix:
server should not crash but report proper errors on the operations that are failing because files cannot be open
[14 May 2008 15:11] Valeriy Kravchuk
FYI, on Windows with 5.1.24 I've got:

C:\Program Files\MySQL\MySQL Server 5.0\bin>mysql -uroot test -P3310 < sel.sql
ERROR 1016 (HY000) at line 1: Can't open file: '.\test\ta306.frm' (errno: 24)

but no server crash.

sel.sql queries MyISAM tables like this:

select * from ta0 UNION
select * from ta1 UNION
select * from ta2 UNION
...
select * from ta2998 UNION
select * from ta2999 UNION
select * from ta3000;

No sure if initial test case was any similar.
[2 Aug 2008 5:17] MySQL Verification Team
davi, valeriy, did you have the general query log enabled ?
[2 Aug 2008 5:33] MySQL Verification Team
this is trivial to repeat but getting a cut and paste testcase is too tricky.

simply set ulimit -n 30 and start a server with
--log --log-output=table --max-connections=2 --table_open_cache=100

then set global max_connections=100
the run a few threads accessing any tables, and run a few flush tables to repopulate the table cache...

080726  6:06:11 [ERROR] Error in accept: Too many open files
080726  6:08:16 [ERROR] /home/sbester/mysql/5.1/mysql-5.1.28-linux-i686/bin/mysqld: Can't open file: './test/t1_tmp.frm' (errno: 24)
080726  6:08:17 [ERROR] /home/sbester/mysql/5.1/mysql-5.1.28-linux-i686/bin/mysqld: Can't open file: './test/t_@002d123.frm' (errno: 24)
Version: '5.1.28-log'  socket: '/tmp/mysql.sock'  port: 3306  Source distribution
080726  6:08:38 [ERROR] /home/sbester/mysql/5.1/mysql-5.1.28-linux-i686/bin/mysqld: Can't open file: './test/t_@002d128.frm' (errno: 24)
/home/sbester/mysql/5.1/mysql-5.1.28-linux-i686/bin/mysqld(my_print_stacktrace+0x3f)[0x835c74f]
/home/sbester/mysql/5.1/mysql-5.1.28-linux-i686/bin/mysqld(handle_segfault+0x3a2)[0x81a3242]
[0xd6d420]
/home/sbester/mysql/5.1/mysql-5.1.28-linux-i686/bin/mysqld(thr_lock+0x4a)[0x835ea1a]
/home/sbester/mysql/5.1/mysql-5.1.28-linux-i686/bin/mysqld(thr_multi_lock+0xc3)[0x835ee53]
/home/sbester/mysql/5.1/mysql-5.1.28-linux-i686/bin/mysqld(mysql_lock_tables(THD*, st_table**, unsigned int, unsigned int, bool*)+0x2be)[0x819e1de]
/home/sbester/mysql/5.1/mysql-5.1.28-linux-i686/bin/mysqld(open_ltable(THD*, TABLE_LIST*, thr_lock_type, unsigned int)+0x198)[0x81ec488]
/home/sbester/mysql/5.1/mysql-5.1.28-linux-i686/bin/mysqld(open_performance_schema_table(THD*, TABLE_LIST*, Open_tables_state*)+0x5a)[0x81ec4fa]
/home/sbester/mysql/5.1/mysql-5.1.28-linux-i686/bin/mysqld(Log_to_csv_event_handler::log_general(THD*, long, char const*, unsigned int, int, char const*, unsigned int, char const*, unsigned int, charset_info_st*)+0xe8)[0x8236768]
/home/sbester/mysql/5.1/mysql-5.1.28-linux-i686/bin/mysqld(LOGGER::general_log_write(THD*, enum_server_command, char const*, unsigned int)+0x165)[0x8234a45]
/home/sbester/mysql/5.1/mysql-5.1.28-linux-i686/bin/mysqld(general_log_write(THD*, enum_server_command, char const*, unsigned int)+0x53)[0x8234b03]
/home/sbester/mysql/5.1/mysql-5.1.28-linux-i686/bin/mysqld(dispatch_command(enum_server_command, THD*, char*, unsigned int)+0x3fe)[0x81b780e]
/home/sbester/mysql/5.1/mysql-5.1.28-linux-i686/bin/mysqld(do_command(THD*)+0xe6)[0x81b8996]
/home/sbester/mysql/5.1/mysql-5.1.28-linux-i686/bin/mysqld(handle_one_connection+0x5c3)[0x81aa513]
/lib/i686/nosegneg/libpthread.so.0[0xde0402]
/lib/i686/nosegneg/libc.so.6(clone+0x5e)[0x20c2ae]
080726  6:08:38 - mysqld got signal 11 ;

thd->query at 0x9e03678 = set @max_val=25
thd->thread_id=11
thd->killed=NOT_KILLED

make sure you get
[2 Aug 2008 5:38] MySQL Verification Team
release version of  mysql-5.1.26-rc-linux-i686-glibc23 crashes too. so this is S1.
[2 Aug 2008 14:54] Davi Arnaut
I was able to reproduce by creating one connection (or thread) that did a couple of selects and sets while other connection was issuing flush tables in a tight loop. Thanks Shane.

The problem seems to be that the CSV storage engine open function returns success even thought it failed to open the data file. Below follows a quick and dirty fix to check whether the crash goes away:

=== modified file 'storage/csv/ha_tina.cc'
--- storage/csv/ha_tina.cc	2008-03-29 15:56:33 +0000
+++ storage/csv/ha_tina.cc	2008-08-02 14:42:39 +0000
@@ -829,7 +829,7 @@ int ha_tina::open(const char *name, int 
 
   local_data_file_version= share->data_file_version;
   if ((data_file= my_open(share->data_file_name, O_RDONLY, MYF(0))) == -1)
-    DBUG_RETURN(0);
+    DBUG_RETURN(1);
 
   /*
     Init locking. Pass handler object to the locking routines,
[2 Aug 2008 14:59] Davi Arnaut
I want to stress the point that the patch is not definitive, far from it, we still need to come up with a better error and do proper cleanup on the error path.
[5 Aug 2008 9:10] Bugs System
A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/50912

2674 Sergey Glukhov	2008-08-05
      Bug#36638 mysqld crashes when open file limit is passed and general query log enabled
      The problem:
      CSV storage engine open function returns success even
      thought it failed to open the data file
      The fix:
      return error
      Additional fixes:
      added MY_WME to my_open to avoid mysterious error message
      free share struct if open the file was unsuccesful
[7 Aug 2008 12:12] Bugs System
A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/51092

2674 Sergey Glukhov	2008-08-07
      Bug#36638 mysqld crashes when open file limit is passed and general query log enabled(2nd ver)
      The problem:
      CSV storage engine open function returns success even
      thought it failed to open the data file
      The fix:
      return error
      Additional fixes:
      added MY_WME to my_open to avoid mysterious error message
      free share struct if open the file was unsuccesful
[15 Aug 2008 7:02] Bugs System
A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/51698

2701 Sergey Glukhov	2008-08-15
      Bug#36638 mysqld crashes when open file limit is passed and general query log enabled
      The problem:
      CSV storage engine open function returns success even
      thought it failed to open the data file
      The fix:
      return error
      Additional fixes:
      added MY_WME to my_open to avoid mysterious error message
      free share struct if open the file was unsuccessful
[15 Aug 2008 7:04] Bugs System
A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/51699

2701 Sergey Glukhov	2008-08-15
      Bug#36638 mysqld crashes when open file limit is passed and general query log enabled
      The problem:
      CSV storage engine open function returns success even
      thought it failed to open the data file
      The fix:
      return error
      Additional fixes:
      added MY_WME to my_open to avoid mysterious error message
      free share struct if open the file was unsuccessful
[19 Aug 2008 13:17] Bugs System
A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/51943

2715 Georgi Kodinov	2008-08-19 [merge]
      Bug#36638: fixing a failing test case.
[19 Aug 2008 13:19] Bugs System
A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/51945

2715 Georgi Kodinov	2008-08-19 [merge]
      Bug#36638: fixing a failing test case.
[21 Aug 2008 18:08] Bugs System
Pushed into 5.1.28  (revid:kgeorge@mysql.com-20080819131620-q59i2vkpmsqhzge0) (version source revid:sergefp@mysql.com-20080819132519-eimtstp3bx89ya9d) (pib:3)
[27 Aug 2008 1:46] Paul DuBois
Noted in 5.1.28 changelog.

The CSV storage engine returned success even when it failed to open a
table's data file.

Setting report to NDI pending push into 6.0.x.
[14 Sep 2008 4:43] Bugs System
Pushed into 6.0.7-alpha  (revid:kgeorge@mysql.com-20080819131620-q59i2vkpmsqhzge0) (version source revid:john.embretsen@sun.com-20080724122511-9c0oudz1xrdrs6y6) (pib:3)
[15 Sep 2008 19:04] Paul DuBois
Noted in 6.0.7 changelog.
[15 Oct 2010 10:10] Kristofer Pettersson
Note:
Test case in 5.1 fails under windows 7.
"select * from t1" produces an empty result set instead of EE_FILENOTFOUND.
Behavior might be sporadic in nature.