MySQL Bugs: #36823: Moved from 3.51 connector to 5.1.4 causing ASP C0000005 errors

Bug #36823	Moved from 3.51 connector to 5.1.4 causing ASP C0000005 errors
Submitted:	20 May 2008 18:30	Modified:	6 Nov 20:20
Reporter:	Richard de Courtney
Status:	Open
Category:	Connector/ODBC	Severity:	S2 (Serious)
Version:	5.1.4	OS:	Microsoft Windows (Server 2003)
Assigned to:	Bogdan Degtyariov	Target Version:
Tags:	c0000005, ASP Error

Description:
After doing a standard parallel install of the ODBC 5.1.4 Connector on our high traffic
site http://FriendSite.com random ASP errors started appearing in our logs. These
errors:

"Error: File /applications/ip2country/lookup.asp  Unexpected error. A trappable error
(C0000005) occurred in an external object. The script cannot continue running.."

This error randomly occurs in various scripts that use the connector. Reverting back to
using the old 3.51 connector stops these errors. I'm guessing that its a memory leak; as
after running IISSTATE, it is showing memory leaks when using the 5.1 ODBC connector.

How to repeat:
Install 5.1.4 ODBC connector on Windows using ASP

Suggested fix:
Fix memory leak?

Hi Richard and thanks for your report.

Based on info (not)provided I do not know where to start... Can you please provide (much)
more info that would help us pinpoint this problem please? 
Start with:
  MySQL server version
  IIS version
  MDAC version
  Database schema and ASP script used
  Are you using x64 OS/driver

If some of info is confidential you can attach it with private flag.

The fact that MyODBC 3.51 works means nothing at this point.

Sorry, I didn't know what other information you needed... here's the info you requested:

MySQL server version: 5.1.24
IIS version: 6
MDAC version: 2.82.3959.0
Database schema and ASP script used: It's happening on any ASP script that's running
standard queries against the DB. We've got 4 mysql databases running, when I change the
ODBC driver from:

objConn.ConnectionString="Driver={MySQL ODBC 3.51
Driver};Server=localhost;Port=33XX;Option=3;Stmt=;Database=XXX;Uid=XXX"

to

objConn.ConnectionString="Driver={MySQL ODBC 5.1
Driver};Server=localhost;Port=33XX;Option=3;Stmt=;Database=XXX;Uid=XXX"

on the scripts, then this is when the random C0000005 starts. Reverting back to the old
driver these stop.

As part of the diagnosis to what was causing the problem, I used IISSTATE which reported
that it detected memory leaks when the scripts were using the 5.1 driver. When I reverted
back to the old driver, no errors were reported.

This is some of the output of the IISSTATE log:

----- START -------

IIS has crashed...
Beginning Analysis
*** WARNING: Unable to verify checksum for C:\Program Files\MySQL\Connector ODBC
5.1\myodbc5.dll
*** ERROR: Symbol file could not be found.  Defaulted to export symbols for C:\Program
Files\MySQL\Connector ODBC 5.1\myodbc5.dll - 
DLL (!FunctionName) that failed: myodbc5!SQLPrepare

Thread ID: 24
System Thread ID: 120c
Kernel Time: 0:1:18.750
User Time: 0:7:14.78
Thread Type: Other 
 # ChildEBP RetAddr  
00 048bd004 00000000 myodbc5!SQLPrepare+0x1c31a
Closing open log file C:\iisstate\output\IISState-4852.log
Opened log file 'C:\iisstate\output\IISState-4852.log'

----- END -------

Are you using x64 OS/driver - no, standard x32 driver on Windows 2003 Standard Edition
Server

Hope that helps, and more than happy to provide any other information you need.

Sorry, I also meant that 3.51 was working, and the only change made was a change in ODBC
driver definition in the db connection script, means that its unlikely to be the ASP
scripts that are at fault for causing the error. As it appears to be a memory leak, then
its not a definitively repeatable step; but we were experiencing the fault every couple
of minutes (with a site that has a few thousand users online at the same time), on some
of the lower used websites, the error was occurring less.

The only difference (that I've just checked and noticed) is that the connections are
pooled for 3.51 (under windows odbc data source administrator) and the 5.1 driver isn't
pooled.

Richard,

We checked myodbc 5.1 driver working in ASP scripts and ADO applications for memory leaks
and could not find any.

Your IIS log shows that the symbol file for myodbc5.dll has not been found.
Please check that myodbc5.pdb (program database) file is in the same directory 
with myodbc5.dll. If there is no such file you can download the MyODBC 5.1 non-install
package and extract myodbc5.pdb from there.
The program database file should provide additional diagnostic information in order to
identify the problem.

Added pdb:

***********************
Starting new log output
IISState version 3.3.1 

Sun May 25 00:29:11 2008

OS = Windows 2003 Server
Executable: w3wp.exe
PID =  2576

Note: Thread times are formatted as HH:MM:SS.ms

***********************

IIS has crashed...
Beginning Analysis
*** WARNING: Unable to verify checksum for C:\Program Files\MySQL\Connector ODBC
5.1\myodbc5.dll
DLL (!FunctionName) that failed: myodbc5!utf16toutf32

Do you have the debugging tools for windows on Windows 2003 Server machine?

Richard,

are you using the debugging tools for windows that provide additional diagnostic for such
errors? If yes, what particular version?
Thanks.

No feedback was provided for this bug for over a month, so it is
being suspended automatically. If you are able to provide the
information that was originally requested, please do so and change
the status of the bug back to "Open".

Can anyone comment on this issue as i am having similar experiences with my ODBC Connector
(v5.01.04.00). 

Similar setup for me:
Windows Server Standard 2003
MySQL v5.1.35-community
MDAC v2.82.3959
Classic ASP application
Connection string: "Driver={MySQL ODBC 5.1 Driver}; Server=localhost; Port=3306;
Option=0; Socket=; Stmt=; Database=[DB]; Uid=[USR]; Pwd=[PWD];"

I'm still having this error as well.. even on clean IIS6/ASP and IIS7/ASP installs. With
it being fully unreproducible (i.e.  no specific script, sql command or components within
the script), it must be a memory leak...?

If someone from MySQL could guide me through debugging, I'd be happy to do this on one of
my social networking sites with high traffic (which should produce those errors pretty
quickly)

Likewise ... i would be willing to throw in some hours and experiment/debug with a medium
traffic production site.

thanks, gw

any progress? anyone willing to comment? had 5 such errors in the last 24hrs.

Guys, we've just given this report higher priority than before so I expect some progress
to be made soon.

I'm getting the same errors having moved from MySQL 5.0/Windows 2003 Server 32bit/3.51
ODBC to MySQL 5.1 64Bit/Windows 2003 Server 64bit/5.1 ODBC.

It seems to occur randomly, both uder ASP & Wscript. For example, we run a simple ASP
script every 60 seconds to test that the system is alive, as follows:

Set ConnData = Server.CreateObject("ADODB.Connection")
ConnData.Open session("indexDSN")
SQL="select field from schema.table"
Set objRst=ConnData.Execute(SQL)
objRst.Close
set ObjRst=nothing
ConnData.Close
set ConnData=nothing

It crashes maybe one in a 1000 times.

64-bit MyODBC driver 5.1.5 was tested by running a simple ASP script for 1 million times.
The memory usage of w3wp.exe process increased from 17Mb to 470Mb. Simple calculations
show that approximately 450 bytes are leaking per call. This has never happened when
running IIS in 32-bit mode with 32-bit myodbc5.dll.

Trying to obtain more information by inserting _CrtDumpMemoryLeaks(); calls in the
entry/exit points of the driver.

Hi everybody,
i have exactly the same behaviour: 3 IIS 6.0 servers crashing at least twice a day each
of them, running several web sites with ASP pages using MyODBC 5.1.5 since august.
Previously we were using MyODBC 3.5 with no problems, but we needed UTF-8 features and
switched to MyODBC 5.1.5. IISState reports always point to : 

DLL (!FunctionName) that failed: myodbc5!SQLPrepare

And :

In 2512-1254828359.dmp the assembly instruction at myodbc5!SQLPrepare+1bfbf in C:\Program
Files\MySQL\Connector ODBC 5.1\myodbc5.dll from MySQL AB has caused an access violation
exception (0xC0000005) when trying to read from memory location 0x0bc12000 on thread 48

Module Information 
Image Name: C:\Program Files\MySQL\Connector ODBC 5.1\myodbc5.dll   
Symbol Type:  Export 
Base address: 0x05400000   
Time Stamp:  Mon Aug 18 13:15:53 2008  
Checksum: 0x00000000   
Comments:  provides core driver functionality 
COM DLL: False   
Company Name:  MySQL AB 
ISAPIExtension: False   
File Description:  MySQL ODBC 5.1 Driver 
ISAPIFilter: False   
File Version:  5, 1, 5, 0 
Managed DLL: False   
Internal Name:  myodbc5 
VB DLL: False   
Legal Copyright:  Copyright © MySQL AB 1995-2006 
Loaded Image Name:  myodbc5.dll   
Legal Trademarks:  MySQL, MyODBC, Connector/ODBC are trademarks of MySQL AB 
Mapped Image Name:     
Original filename:  myodbc5.dll
Module name:  myodbc5   
Private Build:  Production 
Single Threaded:  False   
Product Name:  Connector/ODBC 5.1 
Module Size:  2.26 MBytes   
Product Version:  5, 1, 5, 0 
Symbol File Name:  myodbc5.dll   
Special Build:  &GA release 

I hope you can point out a solution soon.
Thanks,
Juan

After installing 64-bit MDAC on WinXP-64 I found that IIS 6.0 is leaking memory in 64-bit
mode. This is not related to any database activity. A simple ASP script with a counter
reloaded itself 1 million times. w3wp.exe process consumed 470Mb of memory and continued
to get more. This never happened with the same ASP page when running IIS in 32-bit mode.

Checking Win2003 Server x64 Edition...

I don't think it's a 64-bit specific problem. In my case the problem is happening in a
32-bit machine. Anyhow, I've found a possible fix in another bug report:

http://bugs.mysql.com/bug.php?id=44971

I've not tested it yet. Is this compilation reliable?. I don't know, I'll be testing it
tomorrow... If it works, maybe you could push it to snapshot version.

When running ASP script in 32/64 bit mode (Win2003x64 Standard Server) no growing memory
usage was detected after 1M iterations.

Finally I installed the patch and tested it for two days. Though it decreases the number
of "C0000005" errors, they continue occuring at least 6 times a day which is far less
than with the original driver, so at least this lets me work.

Is this bug EVER going to get fixed? :)

John, I am not sure what's there to be fixed, IIS? Please check again on:
 [6 Oct 16:14] Bogdan Degtyariov

After installing 64-bit MDAC on WinXP-64 I found that IIS 6.0 is leaking memory in 64-bit
mode.

Juan, Bug#44971 is fixed and 5.1.6 should be available shortly (if not already).

Finally, we are not paid by the bug report or committed fix or similar. We're truly sorry
when something unrepeatable comes around as we can't fix what we can't reliably
reproduce.

Bogdan, are you ready to close this report?

how can we close this when it it's not even close to being fixed?

First of all, we've all reported no problems with v3.51. The errors begin to appear after
moving to v5.1. Since the rest of the server config remains the same, it must be something
with the newer version. Perhaps it IS iis after all, but the cause of these errors is
created by using v5.1.

Secondly, Bogden reports memory leakage in "iis 6.0" on a windows xp machine. XP comes
with IIS 5.1, not 6. Secondly, he reports no memory leakage in IIS 6 32-&64-bit modes.
Since this O/S is the one we are all using with these problems, perhaps we should
consider it not to be a native memory leak. Perhaps there is something else happening
here. Not knowing enough of the myodbc package, i wouldn't know where to start, but would
be happy to provide any engineer with some of my time any debugging info to assist in
solving this once and for all.

While we're all at the hands of the engineers on this one, we're willing to help out as
best we can. But from my perspective, it's quite repeatable and still open.

And who said anything about getting paid? I've put in lots of my own hours on this issue
...

regards,
 graham

Graham, I just wanted to make sure you understand we do not get any bonus for closing
this, that's all. Also, I want to make sure everybody understands we can not "fix"
something we don't see. It might be apparent and everyday event for you but for us it is
not. Third, it is impossible to trace ODBC calls through IIS which makes things much
harder. Why MS decided to do so is beyond my comprehension...

Leaving it to Bogdan to decide.

Graham,

I use very simple ASP script to determine the version of IIS:

<%
Response.Write(Request.ServerVariables("SERVER_SOFTWARE"))
%>

It prints:

Microsoft-IIS/6.0

Windows XP 32-bit indeed has IIS 5.1, but 64-bit edition is supplied with IIS 6.0.
However, in 64-bit mode it leaks memory even with simple scripts that do not use any
database connectivity. Probably the non-server OS architecture is not good for IIS 6.0.

I just know that we get about 12 C0000005 errors every day running Classic ASP under W2003
Server 64bit - that's not a very helpful bug report I know, but we can reliably replicate
the error in a script, it just happens one in a 1000 times when you open the DSN
connection.

We've tried going back to 3.51 but that also produces "catastrophic" errors more
frequently than the 5.1 C0000005 errors.

And may I say we really do appreciate the time of unpaid folk to look into this!

I may be tempting fate here, but we loaded the new ODBC connector, version 5.1.6
(mysql-connector-odbc-5.1.6-winx64.msi) on Saturday 14-Nov and have had no further erors
since rebooting. This bug wasn't mentioned in the release notes, but either it's been
fixed, gone away or just happens a lot less frequently. Fingers crossed!

In the last release we actually did fix one of causes of C0000005 errors(See Bug#44971,
actually mentioned in ChangeLog as "Random access violation exceptions (0xC0000005) in
ASP scripts...":)). I can't state that was the last or only such bug, but should be one
of most frequently occurring.