| Bug #24509 | 2048 file descriptor limit on windows needs increasing | ||
|---|---|---|---|
| Submitted: | 22 Nov 2006 16:09 | Modified: | 1 Dec 20:44 |
| Reporter: | Shane Bester | ||
| Status: | Closed | ||
| Category: | Server: General | Severity: | S3 (Non-critical) |
| Version: | 5.0,5.1 | OS: | Microsoft Windows (Windows) |
| Assigned to: | Vladislav Vaintroub | Target Version: | 6.0-rc |
| Tags: | bfsm_2006_12_07, fd, 2048, file descriptor, limit, bfsm_2007_02_01 | ||
| Triage: | Triaged: D2 (Serious) | ||
[22 Nov 2006 16:09]
Shane Bester
[10 Dec 2006 11:44]
Shane Bester
btw, I did tests using a small app, 100% win32 api, and found win2003 standard easily opens 45000+ files at a time, and win2003 enterprise went even higher than that.
[20 Dec 2006 20:21]
Shane Bester
see bug #25222 for the win32 handle leak which occurs when _open_osfhandle fails.
[17 Jan 2007 18:17]
Iggy Galarza
I spent some time reading the MSDN information on this problem and there is indeed a hard 2048 file limit when using the _open_osfhandle() function. [See http://msdn2.microsoft.com/en-us/library/kdfaxaay(VS.80).aspx] In order to work around this, the Win32 native HANDLEs should be used instead of the C file handle.
[21 Feb 2007 15:40]
Shane Bester
2048 open posix file limit also exists on 64-bit windows even if binary is built as 64-bit. see attached testcase.
[21 Feb 2007 15:41]
Shane Bester
testcase.
Attachment: fileposix.cpp (text/plain), 844 bytes.
[21 Feb 2007 15:47]
Shane Bester
On my 64-bit XP, the debug binary asserts of you try increase limit > 2048 using _setmaxstdio() call. The only solution is to use 100% win32 handles for file i/o on windows, like innodb does.
[4 Mar 2007 10:09]
Hayden James
any update on this?
[17 Aug 2007 18:00]
Dave Parcels
can someone fix this? i have 100 log entries every day like this.... Could not increase number of max_open_files to more than 2048 (request: 10250) mysql server is crashing all the time during the day when traffic is high. you should know better than to release crappy software like this. these kind of bugs give opensource software a really bad name.
[2 Jan 2008 1:19]
Divya Thakur
This bug makes partitioning unusable on Windows for databases with any significant number of tables/partitions.
[4 Jun 2008 1:09]
Bugs System
A patch for this bug has been committed. After review, it may be pushed to the relevant source trees for release in the next version. You can access the patch from: http://lists.mysql.com/commits/47394 2653 Vladislav Vaintroub 2008-06-04 Bug#24509 - 2048 file descriptor limit on windows needs increasing, also WL#3049 - improved Windows I/O The patch replaces the use of the POSIX I/O interfaces in mysys on Windows with the Win32 API calls (CreateFile, WriteFile, etc). The Windows HANDLE for the open file is stored in the my_file_info struct, along with a flag for append mode (because the Windows API does not support opening files in append mode in all cases) The default max open files has been increased to 16384 and can be increased further by setting --max-open-files=<value> during the server start. Additionally, my_(f)stat() is changed to use __stati64 structure with 64 file size and timestamps. It will now return correct file size now, unlike C runtime stat() that may report outdated info.
[12 Jun 2008 13:20]
Bugs System
A patch for this bug has been committed. After review, it may be pushed to the relevant source trees for release in the next version. You can access the patch from: http://lists.mysql.com/commits/47776 2657 Vladislav Vaintroub 2008-06-12 Bug#24509 - 2048 file descriptor limit on windows needs increasing, also WL#3049 - improved Windows I/O The patch replaces the use of the POSIX I/O interfaces in mysys on Windows with the Win32 API calls (CreateFile, WriteFile, etc). The Windows HANDLE for the open file is stored in the my_file_info struct, along with a flag for append mode (because the Windows API does not support opening files in append mode in all cases) The default max open files has been increased to 16384 and can be increased further by setting --max-open-files=<value> during the server start. Another major change in this patch that almost all Windows specific file IO code has been moved to a new file my_winfile.c, greatly reducing the amount of code in #ifdef blocks within mysys, thus improving readability. Minor enhancements: - my_(f)stat() is changed to use __stati64 structure with 64 file size and timestamps. It will now return correct file size now (C runtime implementation used to report outdated information) - my_lock on Windows is prepared to handle additional timeout parameter
[17 Jun 2008 12:14]
Bugs System
A patch for this bug has been committed. After review, it may be pushed to the relevant source trees for release in the next version. You can access the patch from: http://lists.mysql.com/commits/47983 2657 Vladislav Vaintroub 2008-06-17 Bug#24509 - 2048 file descriptor limit on windows needs increasing, also WL#3049 - improved Windows I/O The patch replaces the use of the POSIX I/O interfaces in mysys on Windows with the Win32 API calls (CreateFile, WriteFile, etc). The Windows HANDLE for the open file is stored in the my_file_info struct, along with a flag for append mode (because the Windows API does not support opening files in append mode in all cases) The default max open files has been increased to 16384 and can be increased further by setting --max-open-files=<value> during the server start. Another major change in this patch that almost all Windows specific file IO code has been moved to a new file my_winfile.c, greatly reducing the amount of code in #ifdef blocks within mysys, thus improving readability. Minor enhancements: - my_(f)stat() is changed to use __stati64 structure with 64 file size and timestamps. It will now return correct file size now (C runtime implementation used to report outdated information) - my_lock on Windows is prepared to handle additional timeout parameter - after review : changed __WIN__ to _WIN32 in the new and changed code.
[18 Jun 2008 0:34]
Bugs System
A patch for this bug has been committed. After review, it may be pushed to the relevant source trees for release in the next version. You can access the patch from: http://lists.mysql.com/commits/48035 2657 Vladislav Vaintroub 2008-06-18 Bug#24509 - 2048 file descriptor limit on windows needs increasing, also WL#3049 - improved Windows I/O The patch replaces the use of the POSIX I/O interfaces in mysys on Windows with the Win32 API calls (CreateFile, WriteFile, etc). The Windows HANDLE for the open file is stored in the my_file_info struct, along with a flag for append mode because the Windows API does not support opening files in append mode in all cases) The default max open files has been increased to 16384 and can be increased further by setting --max-open-files=<value> during the server start. Another major change in this patch that almost all Windows specific file IO code has been moved to a new file my_winfile.c, greatly reducing the amount of code in #ifdef blocks within mysys, thus improving readability. Minor enhancements: - my_(f)stat() is changed to use __stati64 structure with 64 file size and timestamps. It will return correct file size now (C runtime implementation used to report outdated information) - my_lock on Windows is prepared to handle additional timeout parameter - after review : changed __WIN__ to _WIN32 in the new and changed code.
[18 Jun 2008 1:37]
Bugs System
A patch for this bug has been committed. After review, it may be pushed to the relevant source trees for release in the next version. You can access the patch from: http://lists.mysql.com/commits/48036 2657 Vladislav Vaintroub 2008-06-18 Bug#24509 - 2048 file descriptor limit on windows needs increasing, also WL#3049 - improved Windows I/O The patch replaces the use of the POSIX I/O interfaces in mysys on Windows with the Win32 API calls (CreateFile, WriteFile, etc). The Windows HANDLE for the open file is stored in the my_file_info struct, along with a flag for append mode because the Windows API does not support opening files in append mode in all cases) The default max open files has been increased to 16384 and can be increased further by setting --max-open-files=<value> during the server start. Another major change in this patch that almost all Windows specific file IO code has been moved to a new file my_winfile.c, greatly reducing the amount of code in #ifdef blocks within mysys, thus improving readability. Minor enhancements: - my_(f)stat() is changed to use __stati64 structure with 64 file size and timestamps. It will return correct file size now (C runtime implementation used to report outdated information) - my_lock on Windows is prepared to handle additional timeout parameter - after review : changed __WIN__ to _WIN32 in the new and changed code.
[8 Jul 2008 13:26]
Rayan J
Is this fix is coming in the next release 6.0? I am stuck beacuse this is giving problem on merge tables created with union of more than 365*5 tables. (1 table a day) for last 5 yrs.
[8 Jul 2008 15:26]
Vladislav Vaintroub
Ryan, yes, it will come with the 6.0 release(but not necessarily with the next beta)
[17 Oct 2008 18:40]
Bugs System
Pushed into 6.0.8-alpha (revid:vvaintroub@mysql.com-20080617233723-a89vz2gpurlk2p9k) (version source revid:kpettersson@mysql.com-20080915213305-1ljm3tx7tgsdrne9) (pib:5)
[18 Oct 2008 17:31]
Paul DuBois
Noted in 6.0.8 changelog. On WIndows, use of POSIX I/O interfaces in mysys was replaced with Win32 API calls (CreateFile(), WriteFile(), and so forth) and the default maximum number of open files has been increased to 16384. The maximum can be increased further by using the --max-open-files=N option at server startup.
[10 Mar 11:03]
Jeremiah Gowdy
Vladislav, thanks for polishing and integrating my Windows I/O patch. There was supposed to be a one line attribution for my efforts on this in the source code, not sure why that got stripped out, but the code looks good.
[7 Apr 7:59]
Sveta Smirnova
Bug #44122 was marked as duplicate of this one.
[8 Apr 13:43]
Miguel Solorzano
Bug http://bugs.mysql.com/bug.php?id=44147 has been marked as duplicate of this one.
[10 Aug 1:47]
Morgan Tocker
I'm curious if this fix is going to be retargeted to 5.4? Other changes (such as increasing the table_open_cache to 400) will make this limit even more painful.
[10 Aug 1:57]
Vladislav Vaintroub
morgan it *is* in 5.4 already. I know the "target version" 6.0-rc looks strange, but it was planned for 6.0 which became 5.4.
[10 Aug 2:10]
Vladislav Vaintroub
more exactly it is in azalea branch , which is currently numbered as 5.4.4 Thus I say it is in 5.4. But thinking more about this, I do not know what exactly customers will get as 5.4 and whether azalea will be renumbered.
[25 Aug 18:32]
Stefano Zilli
Hi Vladislav, do you know when will be avaible a mysql version for download with this fix ?
[31 Aug 15:26]
Vladislav Vaintroub
@Stefano : I do not know when it comes into an official release. Which features come into which version is something that I unfortunately can not influence. The current "trunk" has been made mostly of 5.1 plus some performance tweaks (google patch etc), so that fix is not planned to be released very soon, but maybe in half year or so. The fix we're talking about lives in mysql-next repository at the moment. If you do not mind building MySQL yourself, you install bzr and get the source with "bzr branch lp:~mysql/mysql-server/mysql-next".
[30 Sep 10:18]
Bugs System
Pushed into 6.0.14-alpha (revid:alik@sun.com-20090929093622-1mooerbh12e97zux) (version source revid:alik@sun.com-20090927210041-r6bfkjhwgcbtyctp) (merge vers: 6.0.14-alpha) (pib:11)
[1 Dec 16:34]
Vladislav Vaintroub
Fixed in 5.5 there is no hard limit on the number of open files anymore.
[1 Dec 16:41]
Stefano Zilli
Hi Vladislav, At what stage is the v5.5 ? I see that v5.4 is still in beta... Stefano
[1 Dec 16:42]
Peter Laursen
6.0 is officially dead. So this will not solve anything for anybody. Will it go into 5.x as well?
[1 Dec 16:53]
Vladislav Vaintroub
@Stefano - it will be in the next beta (which will be 5.6 likely) and then in some future in next release. Note however, that I'm neither directly responsible nor have a clear idea about release/beta ETA, that is I'm not the right person to ask about it.
[1 Dec 17:11]
Stefano Zilli
@Vladislav As the new version and release system is a little bit confusing, could you update us when a public beta with this fix will be avaible? (previously with v5.4 I mean MySql v5.4...)
[1 Dec 20:15]
Vladislav Vaintroub
@Stefano,ok.
[1 Dec 20:44]
Paul DuBois
Noted in 5.5.0 changelog.
[16 Dec 4:11]
Vladislav Vaintroub
Since I promised to inform interested parties, here is the announcement fix is released in 5.5
