| Bug #113554 | ODBC driver 8.1, 8.2, 8.3, 8.4 crashes for large queries | ||
|---|---|---|---|
| Submitted: | 4 Jan 2024 13:08 | Modified: | 20 Aug 2024 23:23 |
| Reporter: | Markus Kollind | Email Updates: | |
| Status: | Closed | Impact on me: | |
| Category: | Connector / ODBC | Severity: | S2 (Serious) |
| Version: | 8.4.0 | OS: | Any |
| Assigned to: | MySQL Verification Team | CPU Architecture: | x86 |
[7 Jan 2024 21:17]
Markus Kollind
The PowerShell function has different names in the definition and the call. Sorry for that. All uses of "Get-ODBC-Query" in the examples should be changed to "Get-ODBC-Data".
[16 Jan 2024 13:59]
Markus Kollind
The issue still exists for the 8.3 driver.
[9 Feb 2024 15:23]
Carl Olsen
I have this exact bug doing queries in a large dataset building plot tables for an analysis. When selecting 5-7 data points, I can get instant crash, when selecting 1-2 data points after 2 or 3 times it crash. Affected version 8.1, 8.2 and 8.3. No issue with 8.0.26
[2 May 2024 9:31]
Markus Kollind
This is still an issue for 8.4.0.
[26 Jul 2024 12:47]
MySQL Verification Team
Hello Markus, Thank you for the bug report. Does it crashes when the connection string does not have these options? "prefetch=1000;NO_CACHE=1" Regards, Ashwini Patil
[12 Aug 2024 11:01]
M S
I have a similar issue, with ACCESS_VIOLATION (C0000005) or a STATUS_HEAP_CORRUPTION (c0000374) happening when my application (64-bit running on Windows 10) is under heavy load, even without prefetch=1000 in connection string (the NO_CACHE=1 seems to have no impact). So I suspect that the prefetch is not the root cause of the crash, it just makes it happen more likely - either that or you have two different memory corruption problems in the driver. So I added running that two SQL queries in bug description to my app, with prefetch=1 in the connection string and added this C++ code before them: _CrtSetDbgFlag(_CRTDBG_ALLOC_MEM_DF | _CRTDBG_DELAY_FREE_MEM_DF | _CRTDBG_CHECK_ALWAYS_DF | _CRTDBG_CHECK_CRT_DF); and now it always crashes in _CrtCheckMemory defined in debug_heap.cpp (on my computer it's located in C:\Program Files (x86)\Windows Kits\10\Source\10.0.22621.0\ucrt\heap) that verifies that memory block headers are correct. Now the variables lead_it and trail_it don't tell anything useful, but I managed to create a natvis debug visualizer for __acrt_first_block that showed entire linked list (until it reaches unreadable memory) and it showed that one of the block (of type _CrtMemBlockHeader) had this binary content: 2c 6d 79 63 6f 6c 75 6d 6e 2c 6d 79 63 6f 6c 75 6d 6e 2c 6d 79 63 6f 6c 75 6d 6e 2c 6d 79 63 6f 6c 75 6d 6e 2c 6d 79 63 6f 6c 75 6d 6e 2c 6d 79 which if I show as ASCII text I get this: ,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,my which is part of the query string. So you have a _huge_ buffer overflow somewhere, that overwrites whole memory blocks. This is a serious backdoor, with severity similar to for example Heartbleed. Until this is fixed, I see the affected ODBC drivers as dangerous to use. This should be fixed ASAP.
[12 Aug 2024 19:17]
Jan Doczy
Hello everyone, I am writing to report a bug I have identified in the scroller_create() function, which I believe is a Use After Free (UAF) issue and is related to this topic. I am not a contributor to this project, but given the simplicity of the fix, I wanted to share the details in the hope that it may be helpful. The bug is located in the scroller_create() function. Specifically, the issue occurs when the stmt->scroller.extend_buf function is called before the query is copied to the buffer. Here is a detailed breakdown of the issue: * Buffer Initialization: By default, the buffer is allocated with a capacity of 1024 bytes. The memory address is stored in stmt->scroller.query, which points to stmt->buf.buf. * Extend Buffer Operation: When the stmt->scroller.extend_buf function is invoked and the query size exceeds the original buffer size (1024), a realloc() call can be indirectly triggered via stmt->buf.extend_buffer. This realloc() call might return a new memory address, different from the one originally allocated. * Problem: The new memory address is not stored back into stmt->scroller.query after the extend_buffer call, leading to the stmt->scroller.query pointer potentially pointing to freed memory. This results in a classic UAF scenario, which can be easily replicated with longer query strings (over 1024 characters). I confirmed this during a debugging session with Application Verifier checks enabled, where the issue was immediately observable. The fix is straightforward: after calling stmt->scroller.extend_buf, the stmt->scroller.query should be updated to point to the potentially new memory address returned by stmt->buf.extend_buffer. This would ensure that the stmt->scroller.query pointer is always valid and points to the correct memory location. Line where first crash is observed: https://github.com/mysql/mysql-connector-odbc/blob/trunk/driver/my_stmt.cc#L607 Query variable is taken from buf.buf as seen in constructor definition: https://github.com/mysql/mysql-connector-odbc/blob/trunk/driver/driver.h#L662 You can see that reallocation of memory is not considered in this line: https://github.com/mysql/mysql-connector-odbc/blob/trunk/driver/driver.h#L667 You can clearly see that extend_buffer is calling realloc here: https://github.com/mysql/mysql-connector-odbc/blob/trunk/driver/utility.cc#L3771 Feel free to contact me for more info. Thanks for fixing this bug!
[13 Aug 2024 9:57]
Bogdan Degtyariov
Hi Jan, Thanks for providing extra details for this issue. It is now verified. In accordance with the company policies we cannot make promises about the timing when the fix is going to be delivered, but it should receive high priority considering the current problem. In our tests the crash happens only when PREFETCH option is used. This is consistent with your observations where scroller buffer is involved. Another temporary workaround would be creating a stored procedure where a long query is executed. Since a workaround exists the severity should be set to S2.
[15 Aug 2024 7:07]
Bogdan Degtyariov
Posted by developer: The problem is fixed by setting the query pointer inside the scroller after buffer reallocation. The patch and the unit test for the issue are pushed to the source tree.
[15 Aug 2024 7:14]
Jan Doczy
Hello Bogdan, Thank you for a quick fix! Can you please also evaluate my other - semi-related finding?: https://bugs.mysql.com/bug.php?id=115829 I believe patching these two issues should solve majority of crashes (we were testing patched versions for a while). Best, Jan
[20 Aug 2024 23:23]
Philip Olson
Fixed as of the upcoming Connector/ODBC 9.1.0 release, and here's the proposed changelog entry from the documentation team: With the prefetch connection option set to a non-zero value, large queries could cause the connector to unexpectedly halt. Thank you Markus for the detailed bug report, and thank you Jan for the thorough analysis.
Description: When sending long queries using the ODBC connector via .NET (6 and 4.8) or PowerShell, the connector sometimes crashes with ACCESS_VIOLATION (C0000005) or a STATUS_HEAP_CORRUPTION (c0000374). We have seen the issue on both linux (with .NET 6) and on windows (with both .NET 6 and .NET Framework 4.8). The same issue exists for both the 8.1 and the 8.2 driver and does not seem to happen every time but quite frequent. It does require that prefetch is enabled, but it does not seem to matter what prefetch is set to (except maybe if you set it really low, e.g., 1, it seems to generate an error rather than a crash). We cannot recreate this with a connection with version before 8.1. How to repeat: We can quite easily recreate this using PowerShell on Windows, using a PowerShell function. These are the steps: 1. Start a docker container with MySQL docker run -d -p 3306:3306 -e MYSQL_ALLOW_EMPTY_PASSWORD=true mysql 2. Open a PowerShell terminal and define the following function: function Get-ODBC-Data{ param( [Parameter(Mandatory)][string]$query, [Parameter(Mandatory)][string]$connstring ) $conn = New-Object System.Data.Odbc.OdbcConnection $conn.ConnectionString = "$connstring" $conn.open() $cmd = New-object System.Data.Odbc.OdbcCommand($query,$conn) $ds = New-Object system.Data.DataSet (New-Object system.Data.odbc.odbcDataAdapter($cmd)).fill($ds) | out-null $conn.close() $ds.Tables[0] } 3. Run the following command (to create something to test on): Get-ODBC-Query -connstring "driver=MySQL ODBC 8.1 ANSI Driver;host=localhost;port=3306;prefetch=1000;NO_CACHE=1;NO_SCHEMA=1;FORWARD_CURSOR=1;DATABASE=mysql;uid=root" -query "create table test (mycolumn varchar(512))" 4. Execute a long dummy-query a couple of times: Get-ODBC-Query -connstring "driver=MySQL ODBC 8.1 ANSI Driver;host=localhost;port=3306;prefetch=1000;NO_CACHE=1;NO_SCHEMA=1;FORWARD_CURSOR=1;DATABASE=mysql;uid=root" -query 'select mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn,mycolumn from test' After two or three times, PowerShell usually crashes with: [process exited with code 3221226356 (0xc0000374)] But sometime we see this error as well: [process exited with code 3221225477 (0xc0000005)] Very rarely it seems like PowerShell do not crash. But starting a new PowerShell session usually results in that it crashes again.