Bug #36823 Moved from 3.51 connector to 5.1.4 causing ASP C0000005 errors
Submitted: 20 May 2008 16:30 Modified: 24 Feb 2010 13:58
Reporter: Richard de Courtney Email Updates:
Status: Closed Impact on me:
None 
Category:Connector / ODBC Severity:S2 (Serious)
Version:5.1.4 OS:Windows (Server 2003)
Assigned to: CPU Architecture:Any
Tags: ASP Error, c0000005

[20 May 2008 16:30] Richard de Courtney
Description:
After doing a standard parallel install of the ODBC 5.1.4 Connector on our high traffic site http://FriendSite.com random ASP errors started appearing in our logs. These errors:

"Error: File /applications/ip2country/lookup.asp  Unexpected error. A trappable error (C0000005) occurred in an external object. The script cannot continue running.."

This error randomly occurs in various scripts that use the connector. Reverting back to using the old 3.51 connector stops these errors. I'm guessing that its a memory leak; as after running IISSTATE, it is showing memory leaks when using the 5.1 ODBC connector.

How to repeat:
Install 5.1.4 ODBC connector on Windows using ASP

Suggested fix:
Fix memory leak?
[20 May 2008 20:25] Tonci Grgin
Hi Richard and thanks for your report.

Based on info (not)provided I do not know where to start... Can you please provide (much) more info that would help us pinpoint this problem please? 
Start with:
  MySQL server version
  IIS version
  MDAC version
  Database schema and ASP script used
  Are you using x64 OS/driver

If some of info is confidential you can attach it with private flag.

The fact that MyODBC 3.51 works means nothing at this point.
[20 May 2008 23:17] Richard de Courtney
Sorry, I didn't know what other information you needed... here's the info you requested:

MySQL server version: 5.1.24
IIS version: 6
MDAC version: 2.82.3959.0
Database schema and ASP script used: It's happening on any ASP script that's running standard queries against the DB. We've got 4 mysql databases running, when I change the ODBC driver from:

objConn.ConnectionString="Driver={MySQL ODBC 3.51 Driver};Server=localhost;Port=33XX;Option=3;Stmt=;Database=XXX;Uid=XXX"

to

objConn.ConnectionString="Driver={MySQL ODBC 5.1 Driver};Server=localhost;Port=33XX;Option=3;Stmt=;Database=XXX;Uid=XXX"

on the scripts, then this is when the random C0000005 starts. Reverting back to the old driver these stop.

As part of the diagnosis to what was causing the problem, I used IISSTATE which reported that it detected memory leaks when the scripts were using the 5.1 driver. When I reverted back to the old driver, no errors were reported.

This is some of the output of the IISSTATE log:

----- START -------

IIS has crashed...
Beginning Analysis
*** WARNING: Unable to verify checksum for C:\Program Files\MySQL\Connector ODBC 5.1\myodbc5.dll
*** ERROR: Symbol file could not be found.  Defaulted to export symbols for C:\Program Files\MySQL\Connector ODBC 5.1\myodbc5.dll - 
DLL (!FunctionName) that failed: myodbc5!SQLPrepare

Thread ID: 24
System Thread ID: 120c
Kernel Time: 0:1:18.750
User Time: 0:7:14.78
Thread Type: Other 
 # ChildEBP RetAddr  
00 048bd004 00000000 myodbc5!SQLPrepare+0x1c31a
Closing open log file C:\iisstate\output\IISState-4852.log
Opened log file 'C:\iisstate\output\IISState-4852.log'

----- END -------

Are you using x64 OS/driver - no, standard x32 driver on Windows 2003 Standard Edition Server

Hope that helps, and more than happy to provide any other information you need.
[20 May 2008 23:25] Richard de Courtney
Sorry, I also meant that 3.51 was working, and the only change made was a change in ODBC driver definition in the db connection script, means that its unlikely to be the ASP scripts that are at fault for causing the error. As it appears to be a memory leak, then its not a definitively repeatable step; but we were experiencing the fault every couple of minutes (with a site that has a few thousand users online at the same time), on some of the lower used websites, the error was occurring less.

The only difference (that I've just checked and noticed) is that the connections are pooled for 3.51 (under windows odbc data source administrator) and the 5.1 driver isn't pooled.
[21 May 2008 21:15] Bogdan Degtyariov
Richard,

We checked myodbc 5.1 driver working in ASP scripts and ADO applications for memory leaks and could not find any.

Your IIS log shows that the symbol file for myodbc5.dll has not been found.
Please check that myodbc5.pdb (program database) file is in the same directory 
with myodbc5.dll. If there is no such file you can download the MyODBC 5.1 non-install package and extract myodbc5.pdb from there.
The program database file should provide additional diagnostic information in order to identify the problem.
[24 May 2008 23:52] Richard de Courtney
Added pdb:

***********************
Starting new log output
IISState version 3.3.1 

Sun May 25 00:29:11 2008

OS = Windows 2003 Server
Executable: w3wp.exe
PID =  2576

Note: Thread times are formatted as HH:MM:SS.ms

***********************

IIS has crashed...
Beginning Analysis
*** WARNING: Unable to verify checksum for C:\Program Files\MySQL\Connector ODBC 5.1\myodbc5.dll
DLL (!FunctionName) that failed: myodbc5!utf16toutf32
[27 May 2008 17:57] Bogdan Degtyariov
Do you have the debugging tools for windows on Windows 2003 Server machine?
[8 Jul 2008 11:07] Bogdan Degtyariov
Richard,

are you using the debugging tools for windows that provide additional diagnostic for such errors? If yes, what particular version?
Thanks.
[8 Aug 2008 23:00] Bugs System
No feedback was provided for this bug for over a month, so it is
being suspended automatically. If you are able to provide the
information that was originally requested, please do so and change
the status of the bug back to "Open".
[28 Jul 2009 17:09] Graham White
Can anyone comment on this issue as i am having similar experiences with my ODBC Connector (v5.01.04.00). 

Similar setup for me:
Windows Server Standard 2003
MySQL v5.1.35-community
MDAC v2.82.3959
Classic ASP application
Connection string: "Driver={MySQL ODBC 5.1 Driver}; Server=localhost; Port=3306; Option=0; Socket=; Stmt=; Database=[DB]; Uid=[USR]; Pwd=[PWD];"
[28 Jul 2009 17:49] Richard de Courtney
I'm still having this error as well.. even on clean IIS6/ASP and IIS7/ASP installs. With it being fully unreproducible (i.e.  no specific script, sql command or components within the script), it must be a memory leak...?

If someone from MySQL could guide me through debugging, I'd be happy to do this on one of my social networking sites with high traffic (which should produce those errors pretty quickly)
[28 Jul 2009 17:59] Graham White
Likewise ... i would be willing to throw in some hours and experiment/debug with a medium traffic production site.

thanks, gw
[11 Aug 2009 15:38] Graham White
any progress? anyone willing to comment? had 5 such errors in the last 24hrs.
[24 Aug 2009 9:58] Tonci Grgin
Guys, we've just given this report higher priority than before so I expect some progress to be made soon.
[16 Sep 2009 11:22] John Paterson
I'm getting the same errors having moved from MySQL 5.0/Windows 2003 Server 32bit/3.51 ODBC to MySQL 5.1 64Bit/Windows 2003 Server 64bit/5.1 ODBC.

It seems to occur randomly, both uder ASP & Wscript. For example, we run a simple ASP script every 60 seconds to test that the system is alive, as follows:

Set ConnData = Server.CreateObject("ADODB.Connection")
ConnData.Open session("indexDSN")
SQL="select field from schema.table"
Set objRst=ConnData.Execute(SQL)
objRst.Close
set ObjRst=nothing
ConnData.Close
set ConnData=nothing

It crashes maybe one in a 1000 times.
[22 Sep 2009 10:34] Bogdan Degtyariov
64-bit MyODBC driver 5.1.5 was tested by running a simple ASP script for 1 million times. The memory usage of w3wp.exe process increased from 17Mb to 470Mb. Simple calculations show that approximately 450 bytes are leaking per call. This has never happened when running IIS in 32-bit mode with 32-bit myodbc5.dll.

Trying to obtain more information by inserting _CrtDumpMemoryLeaks(); calls in the entry/exit points of the driver.
[6 Oct 2009 13:29] Juan F. Capristán W.
Hi everybody,
i have exactly the same behaviour: 3 IIS 6.0 servers crashing at least twice a day each of them, running several web sites with ASP pages using MyODBC 5.1.5 since august. Previously we were using MyODBC 3.5 with no problems, but we needed UTF-8 features and switched to MyODBC 5.1.5. IISState reports always point to : 

DLL (!FunctionName) that failed: myodbc5!SQLPrepare

And :

In 2512-1254828359.dmp the assembly instruction at myodbc5!SQLPrepare+1bfbf in C:\Program Files\MySQL\Connector ODBC 5.1\myodbc5.dll from MySQL AB has caused an access violation exception (0xC0000005) when trying to read from memory location 0x0bc12000 on thread 48

Module Information 
Image Name: C:\Program Files\MySQL\Connector ODBC 5.1\myodbc5.dll   
Symbol Type:  Export 
Base address: 0x05400000   
Time Stamp:  Mon Aug 18 13:15:53 2008  
Checksum: 0x00000000   
Comments:  provides core driver functionality 
COM DLL: False   
Company Name:  MySQL AB 
ISAPIExtension: False   
File Description:  MySQL ODBC 5.1 Driver 
ISAPIFilter: False   
File Version:  5, 1, 5, 0 
Managed DLL: False   
Internal Name:  myodbc5 
VB DLL: False   
Legal Copyright:  Copyright © MySQL AB 1995-2006 
Loaded Image Name:  myodbc5.dll   
Legal Trademarks:  MySQL, MyODBC, Connector/ODBC are trademarks of MySQL AB 
Mapped Image Name:     
Original filename:  myodbc5.dll
Module name:  myodbc5   
Private Build:  Production 
Single Threaded:  False   
Product Name:  Connector/ODBC 5.1 
Module Size:  2.26 MBytes   
Product Version:  5, 1, 5, 0 
Symbol File Name:  myodbc5.dll   
Special Build:  &GA release 

I hope you can point out a solution soon.
Thanks,
Juan
[6 Oct 2009 14:14] Bogdan Degtyariov
After installing 64-bit MDAC on WinXP-64 I found that IIS 6.0 is leaking memory in 64-bit mode. This is not related to any database activity. A simple ASP script with a counter reloaded itself 1 million times. w3wp.exe process consumed 470Mb of memory and continued to get more. This never happened with the same ASP page when running IIS in 32-bit mode.

Checking Win2003 Server x64 Edition...
[6 Oct 2009 14:38] Juan F. Capristán W.
I don't think it's a 64-bit specific problem. In my case the problem is happening in a 32-bit machine. Anyhow, I've found a possible fix in another bug report:

http://bugs.mysql.com/bug.php?id=44971

I've not tested it yet. Is this compilation reliable?. I don't know, I'll be testing it tomorrow... If it works, maybe you could push it to snapshot version.
[9 Oct 2009 7:14] Bogdan Degtyariov
When running ASP script in 32/64 bit mode (Win2003x64 Standard Server) no growing memory usage was detected after 1M iterations.
[9 Oct 2009 10:02] Juan F. Capristán W.
Finally I installed the patch and tested it for two days. Though it decreases the number of "C0000005" errors, they continue occuring at least 6 times a day which is far less than with the original driver, so at least this lets me work.
[6 Nov 2009 16:55] John Paterson
Is this bug EVER going to get fixed? :)
[6 Nov 2009 17:48] Tonci Grgin
John, I am not sure what's there to be fixed, IIS? Please check again on:
 [6 Oct 16:14] Bogdan Degtyariov

After installing 64-bit MDAC on WinXP-64 I found that IIS 6.0 is leaking memory in 64-bit mode.

Juan, Bug#44971 is fixed and 5.1.6 should be available shortly (if not already).

Finally, we are not paid by the bug report or committed fix or similar. We're truly sorry when something unrepeatable comes around as we can't fix what we can't reliably reproduce.

Bogdan, are you ready to close this report?
[6 Nov 2009 18:46] Graham White
how can we close this when it it's not even close to being fixed?

First of all, we've all reported no problems with v3.51. The errors begin to appear after moving to v5.1. Since the rest of the server config remains the same, it must be something with the newer version. Perhaps it IS iis after all, but the cause of these errors is created by using v5.1.

Secondly, Bogden reports memory leakage in "iis 6.0" on a windows xp machine. XP comes with IIS 5.1, not 6. Secondly, he reports no memory leakage in IIS 6 32-&64-bit modes. Since this O/S is the one we are all using with these problems, perhaps we should consider it not to be a native memory leak. Perhaps there is something else happening here. Not knowing enough of the myodbc package, i wouldn't know where to start, but would be happy to provide any engineer with some of my time any debugging info to assist in solving this once and for all.

While we're all at the hands of the engineers on this one, we're willing to help out as best we can. But from my perspective, it's quite repeatable and still open.

And who said anything about getting paid? I've put in lots of my own hours on this issue ...

regards,
 graham
[6 Nov 2009 19:19] Tonci Grgin
Graham, I just wanted to make sure you understand we do not get any bonus for closing this, that's all. Also, I want to make sure everybody understands we can not "fix" something we don't see. It might be apparent and everyday event for you but for us it is not. Third, it is impossible to trace ODBC calls through IIS which makes things much harder. Why MS decided to do so is beyond my comprehension...

Leaving it to Bogdan to decide.
[9 Nov 2009 2:50] Bogdan Degtyariov
Graham,

I use very simple ASP script to determine the version of IIS:

<%
Response.Write(Request.ServerVariables("SERVER_SOFTWARE"))
%>

It prints:

Microsoft-IIS/6.0

Windows XP 32-bit indeed has IIS 5.1, but 64-bit edition is supplied with IIS 6.0. However, in 64-bit mode it leaks memory even with simple scripts that do not use any database connectivity. Probably the non-server OS architecture is not good for IIS 6.0.
[11 Nov 2009 10:36] John Paterson
I just know that we get about 12 C0000005 errors every day running Classic ASP under W2003 Server 64bit - that's not a very helpful bug report I know, but we can reliably replicate the error in a script, it just happens one in a 1000 times when you open the DSN connection.

We've tried going back to 3.51 but that also produces "catastrophic" errors more frequently than the 5.1 C0000005 errors.

And may I say we really do appreciate the time of unpaid folk to look into this!
[17 Nov 2009 13:35] John Paterson
I may be tempting fate here, but we loaded the new ODBC connector, version 5.1.6 (mysql-connector-odbc-5.1.6-winx64.msi) on Saturday 14-Nov and have had no further erors since rebooting. This bug wasn't mentioned in the release notes, but either it's been fixed, gone away or just happens a lot less frequently. Fingers crossed!
[17 Nov 2009 13:58] Lawrenty Novitsky
In the last release we actually did fix one of causes of C0000005 errors(See Bug#44971, actually mentioned in ChangeLog as "Random access violation exceptions (0xC0000005) in ASP scripts...":)). I can't state that was the last or only such bug, but should be one of most frequently occurring.
[24 Feb 2010 13:58] Tonci Grgin
Closing on the basis that the reporter was unable to repeat the problem using newer version.