Bug #59385 mysqld crashes in various ways when starting to run into OS open files limit
Submitted: 10 Jan 2011 5:09 Modified: 7 Dec 2011 16:05
Reporter: Roel Van de Paar Email Updates:
Status: Duplicate Impact on me:
None 
Category:MySQL Server: General Severity:S1 (Critical)
Version:5.5-sec-bk OS:Any
Assigned to: CPU Architecture:Any

[10 Jan 2011 5:09] Roel Van de Paar
Description:
Running a test I got this:

=======================
110110  9:31:24 [Note] /mysql/mysql-5.5-security/mysql-5.5.9-linux-x86_64/bin/mysqld: ready for connections.
Version: '5.5.9-valgrind-max-debug'  socket: '/mysql/mysql-5.5-security/mysql-5.5.9-linux-x86_64/socket-5500.sock'  port: 5500  Source distribution
110110 10:14:11 [ERROR] Found wrong key definition in #sql-31ff_10e; Please do "ALTER TABLE '#sql-31ff_10e' FORCE " to fix it!
110110 10:16:22 [ERROR] /mysql/mysql-5.5-security/mysql-5.5.9-linux-x86_64/bin/mysqld: Can't open file: './test/@0W@0j@1Q@1n@0n@002c@0004.frm' (errno: 24)
110110 10:16:22 [ERROR] /mysql/mysql-5.5-security/mysql-5.5.9-linux-x86_64/bin/mysqld: Can't open file: './test/@0001@0m@0021@008d8@0040@0rj@1k@001a@0026@000e@1P@009b@1PTj@0012@0j@005ewQ@009f@0S@0093@002b@00aa@0095@0020@
00044@008a@0K@00aa.frm' (errno: 24)
110110 10:16:22 [ERROR] /mysql/mysql-5.5-security/mysql-5.5.9-linux-x86_64/bin/mysqld: Can't open file: './test/@00b6@003a@1O5@0018@0024@0g@1P@0025k@00a8O@0023@00ac@0M@00bft@0099I@0K@0093@0j@0pv@0002@0j@1L@0L@001b@0015@0
0bar@00ad@0091I@00b3@0018@0R@003f@009d.frm' (errno: 24)
110110 10:16:22 [ERROR] /mysql/mysql-5.5-security/mysql-5.5.9-linux-x86_64/bin/mysqld: Can't open file: './test/@008f@00b5@000e@0010@0nLt@0001@0088@0z@00a1@0oe@0y@0095@001f@0014.frm' (errno: 24)
110110 10:16:22 [ERROR] /mysql/mysql-5.5-security/mysql-5.5.9-linux-x86_64/bin/mysqld: Can't open file: './test/@005e.frm' (errno: 24)
110110 10:16:22 [ERROR] /mysql/mysql-5.5-security/mysql-5.5.9-linux-x86_64/bin/mysqld: Can't open file: './test/Zq@001fH@1gW@00a0@1gei@003b@00be9f@00b5Wm@00adC.frm' (errno: 24)
110110 10:16:22 [ERROR] /mysql/mysql-5.5-security/mysql-5.5.9-linux-x86_64/bin/mysqld: Can't open file: './test/@005e.frm' (errno: 24)
110110 10:16:22 [ERROR] /mysql/mysql-5.5-security/mysql-5.5.9-linux-x86_64/bin/mysqld: Can't open file: './test/Zq@001fH@1gW@00a0@1gei@003b@00be9f@00b5Wm@00adC.frm' (errno: 24)
110110 10:16:22 [ERROR] /mysql/mysql-5.5-security/mysql-5.5.9-linux-x86_64/bin/mysqld: Can't open file: './test/@005e.frm' (errno: 24)
110110 10:16:22 [ERROR] /mysql/mysql-5.5-security/mysql-5.5.9-linux-x86_64/bin/mysqld: Can't open file: './test/@005e.frm' (errno: 24)
110110 10:16:22 [ERROR] /mysql/mysql-5.5-security/mysql-5.5.9-linux-x86_64/bin/mysqld: Can't open file: './test/@0W@0j@1Q@1n@0n@002c@0004.frm' (errno: 24)
110110 10:16:22 [ERROR] /mysql/mysql-5.5-security/mysql-5.5.9-linux-x86_64/bin/mysqld: Can't open file: './test/@005e.frm' (errno: 24)
110110 10:16:57 [ERROR] Error in accept: Too many open files
110110 10:16:58  InnoDB: Operating system error number 24 in a file operation.
InnoDB: Error number 24 means 'Too many open files'.
InnoDB: Some operating system error numbers are described at
InnoDB: http://dev.mysql.com/doc/refman/5.1/en/operating-system-error-codes.html
InnoDB: File name ./test/@001d.ibd
InnoDB: File operation call: 'open'.
110110 10:16:58  InnoDB: Assertion failure in thread 140296907306752 in file /mysql/mysql-5.5-security/storage
/innobase/fil/fil0fil.c line 803
InnoDB: Failing assertion: ret
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: http://dev.mysql.com/doc/refman/5.1/en/forcing-recovery.html
InnoDB: about forcing recovery.
110110 10:16:58 - mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help diagnose
the problem, but since we have already crashed, something is definitely wrong
and this may fail.

key_buffer_size=8388608
read_buffer_size=131072
max_used_connections=152
max_threads=151
thread_count=147
connection_count=147
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 338633 K
bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0x0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = (nil) thread_stack 0x40000
/mysql/mysql-5.5-security/mysql-5.5.9-linux-x86_64/bin/mysqld(my_print_stacktrace+0x35)[0x8be7ca]
/mysql/mysql-5.5-security/mysql-5.5.9-linux-x86_64/bin/mysqld(handle_segfault+0x31f)[0x554c76]
/lib64/libpthread.so.0[0x32c240f3c0]
/lib64/libc.so.6(gsignal+0x35)[0x32c1c34085]
/lib64/libc.so.6(abort+0x186)[0x32c1c35a36]
/mysql/mysql-5.5-security/mysql-5.5.9-linux-x86_64/bin/mysqld[0x9e1373]
/mysql/mysql-5.5-security/mysql-5.5.9-linux-x86_64/bin/mysqld[0x9e796c]
/mysql/mysql-5.5-security/mysql-5.5.9-linux-x86_64/bin/mysqld[0x9e841a]
/mysql/mysql-5.5-security/mysql-5.5.9-linux-x86_64/bin/mysqld[0x9bb3b8]
/mysql/mysql-5.5-security/mysql-5.5.9-linux-x86_64/bin/mysqld[0x9bb655]
/mysql/mysql-5.5-security/mysql-5.5.9-linux-x86_64/bin/mysqld[0x9bbda7]
/mysql/mysql-5.5-security/mysql-5.5.9-linux-x86_64/bin/mysqld[0x9bc453]
/mysql/mysql-5.5-security/mysql-5.5.9-linux-x86_64/bin/mysqld[0x9bc76a]
/mysql/mysql-5.5-security/mysql-5.5.9-linux-x86_64/bin/mysqld[0x9bc9d7]
/mysql/mysql-5.5-security/mysql-5.5.9-linux-x86_64/bin/mysqld[0x9bcdc6]
/mysql/mysql-5.5-security/mysql-5.5.9-linux-x86_64/bin/mysqld[0x9bcf15]
/mysql/mysql-5.5-security/mysql-5.5.9-linux-x86_64/bin/mysqld[0x9bd4ed]
/mysql/mysql-5.5-security/mysql-5.5.9-linux-x86_64/bin/mysqld[0x9705ee]
/lib64/libpthread.so.0[0x32c2406d5b]
/lib64/libc.so.6(clone+0x6d)[0x32c1ce4a7d]
The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you find out what is causing the crash.
=======================

(More to follow below, but have to split this as bugs system won't allow me to post long descriptions)

How to repeat:
Run many threads creating/altering tables.
[10 Jan 2011 5:10] Roel Van de Paar
Enabled core dumping (did not get a core for the crash above) and restarted mysqld. Got this:

=======================
110110 11:57:05  InnoDB: highest supported file format is Barracuda.
InnoDB: Log scan progressed past the checkpoint lsn 28871809
110110 11:57:05  InnoDB: Database was not shut down normally!
InnoDB: Starting crash recovery.
InnoDB: Reading tablespace information from the .ibd files...
InnoDB: Restoring possible half-written data pages from the doublewrite
InnoDB: buffer...
InnoDB: Doing recovery: scanned up to log sequence number 29232899
InnoDB: 1 transaction(s) which must be rolled back or cleaned up
InnoDB: in total 1 row operations to undo
InnoDB: Trx id counter is 4200
110110 11:57:06  InnoDB: Starting an apply batch of log records to the database...
InnoDB: Progress in percents: 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39
 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 7
6 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99
InnoDB: Apply batch completed
InnoDB: Starting in background the rollback of uncommitted transactions
110110 11:57:08  InnoDB: Rolling back trx with id 4017, 1 rows to undo
110110 11:57:08  InnoDB: Assertion failure in thread 140100795864832 in file /mysql/mysql-5.5-security/storage
/innobase/dict/dict0dict.c line 619
InnoDB: Failing assertion: mutex_own(&dict_sys->mutex)
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: http://dev.mysql.com/doc/refman/5.1/en/forcing-recovery.html
InnoDB: about forcing recovery.
110110 11:57:08 - mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help diagnose
the problem, but since we have already crashed, something is definitely wrong
and this may fail.

key_buffer_size=8388608
read_buffer_size=131072
max_used_connections=0
max_threads=151
thread_count=0
connection_count=0
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 338633 K
bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0x0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = (nil) thread_stack 0x40000
110110 11:57:08  InnoDB: 1.1.4 started; log sequence number 29232899
/mysql/mysql-5.5-security/mysql-5.5.9-linux-x86_64/bin/mysqld(my_print_stacktrace+0x35)[0x8be7ca]
/mysql/mysql-5.5-security/mysql-5.5.9-linux-x86_64/bin/mysqld(handle_segfault+0x31f)[0x554c76]
/lib64/libpthread.so.0[0x32c240f3c0]
/lib64/libc.so.6(gsignal+0x35)[0x32c1c34085]
/lib64/libc.so.6(abort+0x186)[0x32c1c35a36]
/mysql/mysql-5.5-security/mysql-5.5.9-linux-x86_64/bin/mysqld[0x9cf794]
/mysql/mysql-5.5-security/mysql-5.5.9-linux-x86_64/bin/mysqld[0xa56387]
/mysql/mysql-5.5-security/mysql-5.5.9-linux-x86_64/bin/mysqld[0xa56550]
/mysql/mysql-5.5-security/mysql-5.5.9-linux-x86_64/bin/mysqld[0xa4c583]
/mysql/mysql-5.5-security/mysql-5.5.9-linux-x86_64/bin/mysqld[0xa4c692]
/mysql/mysql-5.5-security/mysql-5.5.9-linux-x86_64/bin/mysqld[0xa43839]
/mysql/mysql-5.5-security/mysql-5.5.9-linux-x86_64/bin/mysqld[0xa43a6a]
/mysql/mysql-5.5-security/mysql-5.5.9-linux-x86_64/bin/mysqld[0xa43b8a]
/mysql/mysql-5.5-security/mysql-5.5.9-linux-x86_64/bin/mysqld[0x98142d]
/mysql/mysql-5.5-security/mysql-5.5.9-linux-x86_64/bin/mysqld[0x9817e3]
/mysql/mysql-5.5-security/mysql-5.5.9-linux-x86_64/bin/mysqld[0x9818cb]
/lib64/libpthread.so.0[0x32c2406d5b]
/lib64/libc.so.6(clone+0x6d)[0x32c1ce4a7d]
The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you find out what is causing the crash.
Writing a core file
=======================

gdb bt on that core:

=======================
#0  0x00000032c240c94c in pthread_kill () from /lib64/libpthread.so.0
#1  0x00000000008be856 in my_write_core (sig=6) at /mysql/mysql-5.5-security/mysys/stacktrace.c:423
#2  0x0000000000554f26 in handle_segfault (sig=6) at /mysql/mysql-5.5-security/sql/mysqld.cc:2503
#3  <signal handler called>
#4  0x00000032c1c34085 in raise () from /lib64/libc.so.6
#5  0x00000032c1c35a36 in abort () from /lib64/libc.so.6
#6  0x00000000009cf794 in dict_table_get_on_id (table_id=1, trx=0x20dfad8) at /mysql/mysql-5.5-security/storage/innobase/dict/dict0dict.c:619
#7  0x0000000000a56387 in row_undo_mod_parse_undo_rec (node=0x7f6bac0012b8, thr=0x7f6bac001100) at /mysql/mysql-5.5-security/storage/innobase/row/row0umod.c:813
#8  0x0000000000a56550 in row_undo_mod (node=0x7f6bac0012b8, thr=0x7f6bac001100) at /mysql/mysql-5.5-security/storage/innobase/row/row0umod.c:861
#9  0x0000000000a4c583 in row_undo (node=0x7f6bac0012b8, thr=0x7f6bac001100) at /mysql/mysql-5.5-security/storage/innobase/row/row0undo.c:325
#10 0x0000000000a4c692 in row_undo_step (thr=0x7f6bac001100) at /mysql/mysql-5.5-security/storage/innobase/row/row0undo.c:367
#11 0x0000000000a43839 in que_thr_step (thr=0x7f6bac001100) at /mysql/mysql-5.5-security/storage/innobase/que/que0que.c:1245
#12 0x0000000000a43a6a in que_run_threads_low (thr=0x7f6bac001100) at /mysql/mysql-5.5-security/storage/innobase/que/que0que.c:1305
#13 0x0000000000a43b8a in que_run_threads (thr=0x7f6bac000a00) at /mysql/mysql-5.5-security/storage/innobase/que/que0que.c:1342
#14 0x000000000098142d in trx_rollback_active (trx=0x20dfad8) at /mysql/mysql-5.5-security/storage/innobase/trx/trx0roll.c:472
#15 0x00000000009817e3 in trx_rollback_or_clean_recovered (all=1) at /mysql/mysql-5.5-security/storage/innobase/trx/trx0roll.c:585
#16 0x00000000009818cb in trx_rollback_or_clean_all_recovered (arg=0x7fff4f57181c) at /mysql/mysql-5.5-security/storage/innobase/trx/trx0roll.c:621
#17 0x00000032c2406d5b in start_thread () from /lib64/libpthread.so.0
#18 0x00000032c1ce4a7d in clone () from /lib64/libc.so.6
=======================
[10 Jan 2011 5:10] Roel Van de Paar
Testing some more, ran into this:

=======================
110110 15:20:18 [Note] /mysql/mysql-5.5-security/mysql-5.5.9-linux-x86_64/bin/mysqld: ready for connections.
Version: '5.5.9-valgrind-max-debug'  socket: '/mysql/mysql-5.5-security/mysql-5.5.9-linux-x86_64/socket-5500.s
ock'  port: 5500  Source distribution
/mysql/mysql-5.5-security/mysql-5.5.9-linux-x86_64/bin/mysqld: read looped with error 4, aborting thread
/mysql/mysql-5.5-security/mysql-5.5.9-linux-x86_64/bin/mysqld: read looped with error 4, aborting thread
/mysql/mysql-5.5-security/mysql-5.5.9-linux-x86_64/bin/mysqld: read looped with error 4, aborting thread
/mysql/mysql-5.5-security/mysql-5.5.9-linux-x86_64/bin/mysqld: read looped with error 4, aborting thread
/mysql/mysql-5.5-security/mysql-5.5.9-linux-x86_64/bin/mysqld: read looped with error 4, aborting thread
/mysql/mysql-5.5-security/mysql-5.5.9-linux-x86_64/bin/mysqld: read looped with error 4, aborting thread
/mysql/mysql-5.5-security/mysql-5.5.9-linux-x86_64/bin/mysqld: read looped with error 4, aborting thread
110110 15:41:25 [ERROR] /mysql/mysql-5.5-security/mysql-5.5.9-linux-x86_64/bin/mysqld: Can't open file: './tes
t/@00111e@0p@0091qV@00bc@0q@00a0@0I@1i@0095@0040@007d@1g@003f.frm' (errno: 24)
110110 15:41:25 [ERROR] /mysql/mysql-5.5-security/mysql-5.5.9-linux-x86_64/bin/mysqld: Can't open file: './tes
t/t62.frm' (errno: 24)
110110 15:41:25 [ERROR] /mysql/mysql-5.5-security/mysql-5.5.9-linux-x86_64/bin/mysqld: Can't open file: './tes
t/t81.frm' (errno: 24)
110110 15:41:25 [ERROR] /mysql/mysql-5.5-security/mysql-5.5.9-linux-x86_64/bin/mysqld: Can't open file: './tes
t/@0092@008a@00b4X2@0080f@00b1J@1j@00b8@0094Z.frm' (errno: 24)
110110 15:41:25 [ERROR] /mysql/mysql-5.5-security/mysql-5.5.9-linux-x86_64/bin/mysqld: Can't open file: './tes
t/t11.frm' (errno: 24)
110110 15:41:25 [ERROR] /mysql/mysql-5.5-security/mysql-5.5.9-linux-x86_64/bin/mysqld: Can't open file: './tes
t/t80.frm' (errno: 24)
110110 15:41:25 [ERROR] /mysql/mysql-5.5-security/mysql-5.5.9-linux-x86_64/bin/mysqld: Can't open file: './tes
t/t94.frm' (errno: 24)
110110 15:41:29 [ERROR] /mysql/mysql-5.5-security/mysql-5.5.9-linux-x86_64/bin/mysqld: Can't open file: './tes
t/t31.frm' (errno: 24)
110110 15:41:29 [ERROR] /mysql/mysql-5.5-security/mysql-5.5.9-linux-x86_64/bin/mysqld: Can't open file: './tes
t/t90.frm' (errno: 24)
110110 15:41:29 [ERROR] /mysql/mysql-5.5-security/mysql-5.5.9-linux-x86_64/bin/mysqld: Can't open file: './tes
t/@0h.frm' (errno: 24)
110110 15:41:29 [ERROR] /mysql/mysql-5.5-security/mysql-5.5.9-linux-x86_64/bin/mysqld: Can't open file: './tes
t/t30.frm' (errno: 24)
mysqld: /mysql/mysql-5.5-security/mysys/my_seek.c:57: my_seek: Assertion `fd != -1' failed.
=======================

Ignore the "read looped with error 4, aborting thread" - see bug #34150.

This was running inside gdb. gdb bt:

=======================
#0  0x00000032c1c34085 in raise () from /lib64/libc.so.6
#1  0x00000032c1c35a36 in abort () from /lib64/libc.so.6
#2  0x00000032c1c2c8c5 in __assert_fail () from /lib64/libc.so.6
#3  0x00000000008bac20 in my_seek (fd=-1, pos=0, whence=0, MyFlags=0) at /mysql/mysql-5.5-security/mysys/my_seek.c:57
#4  0x0000000000a65935 in inline_mysql_file_seek (src_file=0xc31c70 "/mysql/mysql-5.5-security/storage/csv/ha_tina.cc", src_line=355, file=-1, pos=0, whence=0, flags=0) at /mysql/mysql-5.5-security/include/mysql/psi/mysql_file.h:1234
#5  0x0000000000a66451 in write_meta_file (meta_file=-1, rows=0, dirty=true) at /mysql/mysql-5.5-security/storage/csv/ha_tina.cc:355
#6  0x0000000000a66798 in free_share (share=0x7ffff000d340) at /mysql/mysql-5.5-security/storage/csv/ha_tina.cc:419
#7  0x0000000000a67a59 in ha_tina::open (this=0x7ffff036f8e0, name=0x7ffff0a249d8 "./test/EP@00a1p@0nYF@0uU@0019D@0096@1h@0095p@00ad@1i@009c@0094@0sG@1L@0R@0089@00a5@00a9@0082@0h@0X@00b9@0013@0k@0O@0x@0k@0o@0v@0092t@000e
@0022@0016@008e@1kS@001b@0V@00d7@0J4@0Q", mode=2, open_options=2) at /mysql/mysql-5.5-security/storage/csv/ha_tina.cc:937
#8  0x0000000000752a66 in handler::ha_open (this=0x7ffff036f8e0, table_arg=0x7ffff0029280, name=0x7ffff0a249d8 "./test/EP@00a1p@0nYF@0uU@0019D@0096@1h@0095p@00ad@1i@009c@0094@0sG@1L@0R@0089@00a5@00a9@0082@0h@0X@00b9@0013@0k@0O@0x@0k@0o@0v@0092t@000e@0022@0016@008e@1kS@001b@0V@00d7@0J4@0Q", mode=2, test_if_locked=2) at /mysql/mysql-5.5-security/sql/handler.cc:2141
#9  0x000000000069a6e1 in open_table_from_share (thd=0x208fac0, share=0x7ffff0a24610, alias=0x7ffff0105810 "EP¡pçYFîU\031D\302\226õ\302\225p­ö\302\234\302\224ìGÙË\302\211¥©\302\202áѹ\023äÈñäèï\302\222t\016\"\026\302\216
øS\033Ï×Ã4Ê", db_stat=39, prgflag=44, ha_open_flags=0, outparam=0x7ffff0029280, is_create_table=false) at /mysql/mysql-5.5-security/sql/table.cc:2003
#10 0x00000000005a1cd6 in open_table (thd=0x208fac0, table_list=0x7ffff08d2bf0, mem_root=0x7fffe29abf60, ot_ctx=0x7fffe29abfa0) at /mysql/mysql-5.5-security/sql/sql_base.cc:3033
#11 0x00000000005a3f5d in open_and_process_table (thd=0x208fac0, lex=0x20916e0, tables=0x7ffff08d2bf0, counter=0x7fffe29ac0c4, flags=0, prelocking_strategy=0x7fffe29ad720, has_prelocking_list=false, ot_ctx=0x7fffe29abfa0
, new_frm_mem=0x7fffe29abf60) at /mysql/mysql-5.5-security/sql/sql_base.cc:4362
#12 0x00000000005a4cb0 in open_tables (thd=0x208fac0, start=0x7fffe29ac080, counter=0x7fffe29ac0c4, flags=0, prelocking_strategy=0x7fffe29ad720) at /mysql/mysql-5.5-security/sql/sql_base.cc:4802
#13 0x00000000005a5b44 in open_and_lock_tables (thd=0x208fac0, tables=0x7ffff08d2bf0, derived=false, flags=0,
prelocking_strategy=0x7fffe29ad720) at /mysql/mysql-5.5-security/sql/sql_base.cc:5399
#14 0x0000000000677c51 in mysql_alter_table (thd=0x208fac0, new_db=0x7ffff0022be0 "test", new_name=0x0, create_info=0x7fffe29ade50, table_list=0x7ffff08d2bf0, alter_info=0x7fffe29adf30, order_num=0, order=0x0, ignore=tru
e) at /mysql/mysql-5.5-security/sql/sql_table.cc:5756
#15 0x000000000089552f in Alter_table_statement::execute (this=0x7ffff0351550, thd=0x208fac0) at /mysql/mysql-5.5-security/sql/sql_alter.cc:106
#16 0x00000000005f7462 in mysql_execute_command (thd=0x208fac0) at /mysql/mysql-5.5-security/sql/sql_parse.cc:4321
#17 0x00000000005f9eac in mysql_parse (thd=0x208fac0, rawbuf=0x7ffff0036580 "alter ignore table `test`.`EP¡pçYFîU\031D\302\226õ\302\225p­ö\302\234\302\224ìGÙË\302\211¥©\302\202áѹ\023äÈñäèï\302\222t\016\"\026\302\216øS\0
33Ï×Ã4Ê` add column `col78` tinyint zerofill not null", length=157, parser_state=0x7fffe29af630) at /mysql/mysql-5.5-security/sql/sql_parse.cc:5509
#18 0x00000000005ee3ef in dispatch_command (command=COM_QUERY, thd=0x208fac0, packet=0x20b2291 "alter ignore table `test`.`EP¡pçYFîU\031D\302\226õ\302\225p­ö\302\234\302\224ìGÙË\302\211¥©\302\202áѹ\023äÈñäèï\302\222t\01
6\"\026\302\216øS\033Ï×Ã4Ê` add column `col78` tinyint zerofill not null ", packet_length=158) at /mysql/mysql-5.5-security/sql/sql_parse.cc:1035
#19 0x00000000005ed725 in do_command (thd=0x208fac0) at /mysql/mysql-5.5-security/sql/sql_parse.cc:772#20 0x00000000006c9ad7 in do_handle_one_connection (thd_arg=0x208fac0) at /mysql/mysql-5.5-security/sql/sql_connect.cc:748
#21 0x00000000006c9668 in handle_one_connection (arg=0x208fac0) at /mysql/mysql-5.5-security/sql/sql_connect.cc:684
#22 0x00000032c2406d5b in start_thread () from /lib64/libpthread.so.0
#23 0x00000032c1ce4a7d in clone () from /lib64/libc.so.6
=======================

A similar stacktrace is listed here: bug #54486 - but notice in the corresponding error log the same 24 (too many open files) error...
[10 Jan 2011 7:11] MySQL Verification Team
The CSV engine crash looks like bug #57895
[10 Jan 2011 7:21] Roel Van de Paar
[root@mac mysql-5.5.9-linux-x86_64]# ulimit -n
1024
[root@mac mysql-5.5.9-linux-x86_64]# cat my.cnf
[mysqld]
core-file
basedir=/mysql/mysql-5.5-security/mysql-5.5.9-linux-x86_64/
socket=/mysql/mysql-5.5-security/mysql-5.5.9-linux-x86_64/socket-5500.sock
pid=/mysql/mysql-5.5-security/mysql-5.5.9-linux-x86_64/pid-5500.pid
port=5500
user=root
log-bin-trust-function-creators
character-set-server=latin1
log-error=/mysql/mysql-5.5-security/mysql-5.5.9-linux-x86_64/data/error.log
[10 Jan 2011 7:45] Roel Van de Paar
More issues/crash after restarting mysqld and continuing to run some threads - lots of different errors.

Attachment: error.log (application/octet-stream, text), 191.04 KiB.

[10 Jan 2011 7:55] Roel Van de Paar
Corresponding (with latter part in error log) gdb output

(gdb) bt
#0  0x00000032c1c34085 in raise () from /lib64/libc.so.6
#1  0x00000032c1c35a36 in abort () from /lib64/libc.so.6
#2  0x00000032c1c2c8c5 in __assert_fail () from /lib64/libc.so.6
#3  0x00000000008bac20 in my_seek (fd=-1, pos=0, whence=0, MyFlags=0)
    at /mysql/mysql-5.5-security/mysys/my_seek.c:57
#4  0x0000000000a65935 in inline_mysql_file_seek (
    src_file=0xc31c70 "/mysql/mysql-5.5-security/storage/csv/ha_tina.cc", src_line=355, file=-1, pos=0,
    whence=0, flags=0) at /mysql/mysql-5.5-security/include/mysql/psi/mysql_file.h:1234
#5  0x0000000000a66451 in write_meta_file (meta_file=-1, rows=0, dirty=true)
    at /mysql/mysql-5.5-security/storage/csv/ha_tina.cc:355
#6  0x0000000000a66798 in free_share (share=0x7ffff012ac50)
    at /mysql/mysql-5.5-security/storage/csv/ha_tina.cc:419
#7  0x0000000000a67a59 in ha_tina::open (this=0x7ffff017d3a0, name=0x7ffff01671f0 "./test/#sql-2faa_216",
    mode=2, open_options=2) at /mysql/mysql-5.5-security/storage/csv/ha_tina.cc:937
#8  0x0000000000752a66 in handler::ha_open (this=0x7ffff017d3a0, table_arg=0x7ffff01665d0,
    name=0x7ffff01671f0 "./test/#sql-2faa_216", mode=2, test_if_locked=2)
    at /mysql/mysql-5.5-security/sql/handler.cc:2141
#9  0x000000000069a6e1 in open_table_from_share (thd=0x20387a0, share=0x7ffff0166e88,
    alias=0x7fffd5074680 "#sql-2faa_216", db_stat=7, prgflag=44, ha_open_flags=0, outparam=0x7ffff01665d0,
    is_create_table=false) at /mysql/mysql-5.5-security/sql/table.cc:2003
#10 0x00000000005a69c9 in open_table_uncached (thd=0x20387a0, path=0x7fffd5073690 "./test/#sql-2faa_216",
    db=0x7ffff0043c40 "test", table_name=0x7fffd5074680 "#sql-2faa_216", add_to_temporary_tables_list=true)
    at /mysql/mysql-5.5-security/sql/sql_base.cc:5816
#11 0x00000000006794b9 in mysql_alter_table (thd=0x20387a0, new_db=0x7ffff0043c40 "test",
    new_name=0x7ffff0069570 "t34", create_info=0x7fffd5074e50, table_list=0x7ffff0127f80,
    alter_info=0x7fffd5074f30, order_num=0, order=0x0, ignore=false)
    at /mysql/mysql-5.5-security/sql/sql_table.cc:6370
#12 0x000000000089552f in Alter_table_statement::execute (this=0x7ffff002e570, thd=0x20387a0)
    at /mysql/mysql-5.5-security/sql/sql_alter.cc:106
#13 0x00000000005f7462 in mysql_execute_command (thd=0x20387a0)
    at /mysql/mysql-5.5-security/sql/sql_parse.cc:4321
#14 0x00000000005f9eac in mysql_parse (thd=0x20387a0,
    rawbuf=0x7ffff00f7040 "alter table `test`.`t34` add column `col61` blob not null", length=57,
    parser_state=0x7fffd5076630) at /mysql/mysql-5.5-security/sql/sql_parse.cc:5509
#15 0x00000000005ee3ef in dispatch_command (command=COM_QUERY, thd=0x20387a0,
    packet=0x1fa2e21 "alter table `test`.`t34` add column `col61` blob not null ", packet_length=58)
    at /mysql/mysql-5.5-security/sql/sql_parse.cc:1035
#16 0x00000000005ed725 in do_command (thd=0x20387a0) at /mysql/mysql-5.5-security/sql/sql_parse.cc:772
#17 0x00000000006c9ad7 in do_handle_one_connection (thd_arg=0x20387a0)
    at /mysql/mysql-5.5-security/sql/sql_connect.cc:748
#18 0x00000000006c9668 in handle_one_connection (arg=0x20387a0)
    at /mysql/mysql-5.5-security/sql/sql_connect.cc:684
#19 0x00000032c2406d5b in start_thread () from /lib64/libpthread.so.0
#20 0x00000032c1ce4a7d in clone () from /lib64/libc.so.6

(gdb) continue
Continuing.
[New Thread 0x7fffe44ec700 (LWP 19418)]
[New Thread 0x7fffd65b3700 (LWP 19419)]

Program received signal SIGABRT, Aborted.
0x00000032c240c94c in pthread_kill () from /lib64/libpthread.so.0

(gdb) bt
#0  0x00000032c240c94c in pthread_kill () from /lib64/libpthread.so.0
#1  0x00000000008be856 in my_write_core (sig=6) at /mysql/mysql-5.5-security/mysys/stacktrace.c:423
#2  0x0000000000554f26 in handle_segfault (sig=6) at /mysql/mysql-5.5-security/sql/mysqld.cc:2503
#3  <signal handler called>
#4  0x00000032c1c34085 in raise () from /lib64/libc.so.6
#5  0x00000032c1c35a36 in abort () from /lib64/libc.so.6
#6  0x00000032c1c2c8c5 in __assert_fail () from /lib64/libc.so.6
#7  0x00000000008bac20 in my_seek (fd=-1, pos=0, whence=0, MyFlags=0)
    at /mysql/mysql-5.5-security/mysys/my_seek.c:57
#8  0x0000000000a65935 in inline_mysql_file_seek (
    src_file=0xc31c70 "/mysql/mysql-5.5-security/storage/csv/ha_tina.cc", src_line=355, file=-1, pos=0,
    whence=0, flags=0) at /mysql/mysql-5.5-security/include/mysql/psi/mysql_file.h:1234
#9  0x0000000000a66451 in write_meta_file (meta_file=-1, rows=0, dirty=true)
    at /mysql/mysql-5.5-security/storage/csv/ha_tina.cc:355
#10 0x0000000000a66798 in free_share (share=0x7ffff012ac50)
    at /mysql/mysql-5.5-security/storage/csv/ha_tina.cc:419
#11 0x0000000000a67a59 in ha_tina::open (this=0x7ffff017d3a0, name=0x7ffff01671f0 "./test/#sql-2faa_216",
    mode=2, open_options=2) at /mysql/mysql-5.5-security/storage/csv/ha_tina.cc:937
#12 0x0000000000752a66 in handler::ha_open (this=0x7ffff017d3a0, table_arg=0x7ffff01665d0,
    name=0x7ffff01671f0 "./test/#sql-2faa_216", mode=2, test_if_locked=2)
    at /mysql/mysql-5.5-security/sql/handler.cc:2141
#13 0x000000000069a6e1 in open_table_from_share (thd=0x20387a0, share=0x7ffff0166e88,
    alias=0x7fffd5074680 "#sql-2faa_216", db_stat=7, prgflag=44, ha_open_flags=0, outparam=0x7ffff01665d0,
    is_create_table=false) at /mysql/mysql-5.5-security/sql/table.cc:2003
#14 0x00000000005a69c9 in open_table_uncached (thd=0x20387a0, path=0x7fffd5073690 "./test/#sql-2faa_216",
    db=0x7ffff0043c40 "test", table_name=0x7fffd5074680 "#sql-2faa_216", add_to_temporary_tables_list=true)
    at /mysql/mysql-5.5-security/sql/sql_base.cc:5816
#15 0x00000000006794b9 in mysql_alter_table (thd=0x20387a0, new_db=0x7ffff0043c40 "test",
    new_name=0x7ffff0069570 "t34", create_info=0x7fffd5074e50, table_list=0x7ffff0127f80,
    alter_info=0x7fffd5074f30, order_num=0, order=0x0, ignore=false)
    at /mysql/mysql-5.5-security/sql/sql_table.cc:6370
#16 0x000000000089552f in Alter_table_statement::execute (this=0x7ffff002e570, thd=0x20387a0)
    at /mysql/mysql-5.5-security/sql/sql_alter.cc:106
#17 0x00000000005f7462 in mysql_execute_command (thd=0x20387a0)
    at /mysql/mysql-5.5-security/sql/sql_parse.cc:4321
#18 0x00000000005f9eac in mysql_parse (thd=0x20387a0,
    rawbuf=0x7ffff00f7040 "alter table `test`.`t34` add column `col61` blob not null", length=57,
    parser_state=0x7fffd5076630) at /mysql/mysql-5.5-security/sql/sql_parse.cc:5509
#19 0x00000000005ee3ef in dispatch_command (command=COM_QUERY, thd=0x20387a0,
    packet=0x1fa2e21 "alter table `test`.`t34` add column `col61` blob not null ", packet_length=58)
    at /mysql/mysql-5.5-security/sql/sql_parse.cc:1035
#20 0x00000000005ed725 in do_command (thd=0x20387a0) at /mysql/mysql-5.5-security/sql/sql_parse.cc:772
#21 0x00000000006c9ad7 in do_handle_one_connection (thd_arg=0x20387a0)
    at /mysql/mysql-5.5-security/sql/sql_connect.cc:748
#22 0x00000000006c9668 in handle_one_connection (arg=0x20387a0)
    at /mysql/mysql-5.5-security/sql/sql_connect.cc:684
#23 0x00000032c2406d5b in start_thread () from /lib64/libpthread.so.0
#24 0x00000032c1ce4a7d in clone () from /lib64/libc.so.6
[10 Jan 2011 7:57] Roel Van de Paar
Btw, the InnoDB "has waited ... seconds the semaphore" can likely be ignored as they may have been the result of the gdb pause (after which continue was executed)
[10 Jan 2011 8:27] Roel Van de Paar
From Marko: The “110110 10:16:57 [ERROR] Error in accept: Too many open files” is a MySQL error, not an InnoDB error. Accept is the call to accept a TCP/IP socket connection, and that one will attempt allocate a file descriptor as well. The “110110 10:16:58  InnoDB: Operating system error number 24 in a file operation.” a second later is the same error reported by InnoDB (i.e., InnoDB complains when it attempts to open the *.ibd file a second after the accept() failed in MYSQL).
[11 Jan 2011 22:23] Omer Barnir
triage: setting tag to SR55MRU (w3 - increase the limit of open files
[10 Feb 2011 9:39] Marko Mäkelä
As far as InnoDB is concerned, the problem is that InnoDB does not do graceful error handling when a read fails. (Write failures should probably result in a server crash, but read failures could be handled by denying access to the affected data.)

The InnoDB fix for this could be a small part of Bug #10132 Crashing the server on corrupt InnoDB page is unhelpful