MySQL Bugs: #115678: Debugging doesn't work with provided PDB file

Bug #115678	Debugging doesn't work with provided PDB file
Submitted:	24 Jul 8:57	Modified:	24 Jul 14:32
Reporter:	peter janovic	Email Updates:
Status:	Closed	Impact on me:	None
Category:	MySQL Server: C API (client library)	Severity:	S6 (Debug Builds)
Version:	9.0 8.0.37	OS:	Windows (11 latest)
Assigned to:		CPU Architecture:	x86
Tags:	debugging, debugging symbols

Description:
I've been coding against the MySQL C API client library for the last ~2 years. I tried to set up debugging for libmysql.dll Release builds with the PDB file provided at https://dev.mysql.com/downloads/mysql/ (Windows (x86, 64-bit), ZIP Archive
Debug Binaries & Test Suite).

But, I always failed, I have never been able to Step in or set breakpoints using this libmysql.pdb file. And I tried hard, last time I invested ~4 hours into it.

I tried both QtCreator and also Visual Studio 2022.

I never knew why it doesn't work, until now. The PDB file provided isn't working because of the PDB signature mismatch.

Here is the output of dumpbin /PDBPATH:VERBOSE .\libmysql.dll

Dump of file .\libmysql.dll

File Type: DLL
  PDB file 'C:\Program Files\MySQL\MySQL Server 9.0\lib\libmysql.pdb' checked.  (PDB signature mismatch)
  PDB file 'C:\build\sb_1-15774161-1717764653.26\release\library_output_directory\RelWithDebInfo\libmysql.pdb' checked.  (File not found)
  PDB file 'C:\WINDOWS\symbols\dll\libmysql.pdb' checked.  (File not found)
  PDB file 'C:\WINDOWS\dll\libmysql.pdb' checked.  (File not found)
  PDB file 'C:\WINDOWS\libmysql.pdb' checked.  (File not found)

And here is another output using the ChkMatch.exe -c .\libmysql.dll .\libmysql.pdb

ChkMatch - version 1.0
Copyright (C) 2004 Oleg Starodumov
http://www.debuginfo.com/

Executable: .\libmysql.dll
Debug info file: .\libmysql.pdb

Executable:
TimeDateStamp: 66630da9
Debug info: 2 ( CodeView )
TimeStamp: 66630da9  Characteristics: 0  MajorVer: 0  MinorVer: 0
Size: 122  RVA: 003c7d28  FileOffset: 003c6f28
CodeView format: RSDS
Signature: {f6a5eb70-03b1-4232-9be2-49db165405b0}  Age: 1
PdbFile: C:\build\sb_1-15774161-1717764653.26\release\library_output_directory\RelWithDebInfo\libmysql.pdb
Debug info: 12 ( Unknown )
TimeStamp: 66630da9  Characteristics: 0  MajorVer: 0  MinorVer: 0
Size: 20  RVA: 003c7da4  FileOffset: 003c6fa4
Debug info: 13 ( Unknown )
TimeStamp: 66630da9  Characteristics: 0  MajorVer: 0  MinorVer: 0
Size: 920  RVA: 003c7db8  FileOffset: 003c6fb8

Debug information file:
Format: PDB 7.00
Signature: {0949abd5-aaeb-4543-9529-d8f98a631a72}  Age: 1

Result: Unmatched (reason: Signature mismatch)

I'm practically coding blindly against the MySQL C API.

If I really have a bug that I must debug and I need to Step into the MySQL C API code, then I do the vcpkg libmysql port build and in this case, I'm able to debug properly. But this isn't for everyday use, of course, it's only in case of emergency.

This problem has been around for as long as I can remember in MySQL 8.

How to repeat:
--

MSVC Toolchain used is always the latest stable.

Eg. here is the simplest Hello World example where I should be able to Step Into the mysql_init() in the QtCreator and Visual Studio 2022.

#include <iostream>
#include <mysql.h>

int main()
{
	auto *mysql = mysql_init(nullptr);

	std::cout << "Hello World" << std::endl;
	return 0;
}

HI Mr. Janovic,

Thank you for your bug report.

We truly do not see what is your problem.

First of all, there is no need to make breakpoints into libmysql at all. It is practically free of bugs and you should look into your code only. Most likely, you would only be confused by inspecting our libmysql code.

Next, you should use our static library, instead of the DLL.

We wonder why haven't you gone with the most obvious solution for your problem. You could have tried to build debug version of our static libmysql library.

Please, try all the above, while we contact our Release Management about the above mismatch that you are reporting.

We are waiting on your feedback.

I'm not confused about code in MySQL C API source code 😁 and I'm actually using the source code very often to better understand many things. Eg. my last good hit was that the MySQL C API appends '\0' during mysql_stmt_bind_result() for the MYSQL_TYPE_VAR_STRING if the buffer is long enough.

All yours provided solutions don't change the fact that the PDB files are wrong and don't work.

Now I see you tagged it as Can't repeat, are you able to use this libmysql.pdb? Am I wrong?

Also to be exact I'm talking about the libmysql.pdb file only, I didn't test other PDB files because I don't need them.

Hi,

We are leaving this report in another status, because we are waiting for the feedback from the team in charge of building those packages.

But, really analysing code in our C API library is fine.

What we truly do not understand is why do you have to debug it.

Debugging through our C API code will not help you find your own error. For that purpose, we have a documentation for our C API and you will be able to concentrate on your own bugs, instead on our bugs. At this moment, we do not have any outstanding bugs in our C API.

Can't you just concentrate on debugging your own code ???

HI Mr. Janovic,

While we are all waiting on the feedback from the team in charge, one word of the free advice.

You can download our source code ZIP file and inspect the file named mysql_client_test.cc.

It is in the testclients/ directory.

It has so many finely coded  tests, so that you can use them for your application.

FWIW, I've been able to step into libmysql.dll's mysql_init() using the binaries provided in https://dev.mysql.com/downloads/mysql/.

Here's what I did:

1. I've downloaded (mysql-8.0.39-winx64.zip) MD5: acee83dd1f2fbd0317cf0cee9ddc4f10 and have extracted it into C:\Users\gkodinov\Downloads\mysql-8.0.39-winx64
2. I've downloaded (mysql-8.0.39-winx64-debug-test.zip) MD5: 20be7cbf369c241fa2e9754a66ff8cf2 and have extracted it into C:\Users\gkodinov\Downloads\mysql-8.0.39-winx64-debug-test
3. Using VI I've created a file called x.cc in C:\Users\gkodinov\Downloads\:
C:\Users\gkodinov\Downloads>type x.cc
#include <iostream>
#include <mysql.h>

int main()
{
auto *mysql = mysql_init(nullptr);

std::cout << "Hello World" << std::endl;
return 0;
}

4. I've compiled the source from a Visual Studio 2022 Developer command prompt terminal as follows:
cl /Zi x.cc /I mysql-8.0.39-winx64\include mysql-8.0.39-winx64\lib\libmysql.lib
5. I've copied libmysql.dll into the current directory:
copy mysql-8.0.39-winx64\lib\libmysql.dll .
6. I've started x.exec in devenv as follows:
devenv /debugexe x.exe
7. In devenv I've opened Tools/Options/Symbols and have added the full path "C:\Users\gkodinov\Downloads\mysql-8.0.39-winx64-debug-test\mysql-8.0.39-winx64\lib" to the Symbol file locations list (last entry).
8. I've pressed F11 (step into) and that (after some time of caching the .pdb files for various dlls and binaries) got x.cc opened and my cursor on the opening curly brace.
9. I've stepped to the mysql_init() call using F10 (step over).
10. I've pressed F11 (step into) and got the window called "client.cc not found" and a choice to point to it manually or to view disassembly. Both worked.

Please try the above sequence. It has worked for me.
My VS2022 version is:

Microsoft Visual Studio Professional 2022
Version 17.10.4
VisualStudio.17.Release/17.10.4+35027.167
Microsoft .NET Framework
Version 4.8.09032

Installed Version: Professional

Visual C++ 2022 00483-10801-26043-AA846
Microsoft Visual C++ 2022

ASP.NET and Web Tools 17.10.344.38934
ASP.NET and Web Tools

Azure App Service Tools v3.0.0 17.10.344.38934
Azure App Service Tools v3.0.0

C# Tools 4.10.0-3.24324.8+04fb39164c99c519832109f21b22630bc9246fce
C# components used in the IDE. Depending on your project type and settings, a different version of the compiler may be used.

GitHub Copilot 0.2.889.30432
GitHub Copilot is an AI pair programmer that helps you write code faster and with less work.

Linux Core Dump Debugging 1.0.9.35027
Enables debugging of Linux core dumps.

Microsoft JVM Debugger 1.0
Provides support for connecting the Visual Studio debugger to JDWP compatible Java Virtual Machines

NuGet Package Manager 6.10.1
NuGet Package Manager in Visual Studio. For more information about NuGet, visit https://docs.nuget.org/

StopOnFirstBuildError 1.0
StopOnFirstBuildError

Stops a solution build immediately after a project has failed to build.

https://einaregilsson.com/stop-build-on-first-error-in-visual-studio-2010/

Test Adapter for Boost.Test 1.0
Enables Visual Studio's testing tools with unit tests written for Boost.Test. The use terms and Third Party Notices are available in the extension installation directory.

Test Adapter for Google Test 1.0
Enables Visual Studio's testing tools with unit tests written for Google Test. The use terms and Third Party Notices are available in the extension installation directory.

TypeScript Tools 17.0.30327.2001
TypeScript Tools for Microsoft Visual Studio

Visual Basic Tools 4.10.0-3.24324.8+04fb39164c99c519832109f21b22630bc9246fce
Visual Basic components used in the IDE. Depending on your project type and settings, a different version of the compiler may be used.

Visual C++ for Linux Development 1.0.9.35027
Visual C++ for Linux Development

Visual Studio IntelliCode 2.2
AI-assisted development for Visual Studio.

Thank you, Mr. Kodinov.

Hence, we can not repeat it ......

HI Mr. Janovic,

We also have a message from our Build team:

The text in Bug #115678 shows the path to the libmysql.pdb as C:\Program Files\MySQL\MySQL Server 9.0\lib\libmysql.pdb, so I've just checked the libmysql.dll file extracted from mysql-9.0.1-winx64.zip (downloaded from the MySQL community downloads site https://dev.mysql.com/downloads) against the libmysql.pdb file extracted from mysql-9.0.1-winx64-debug-test.zip (also downloaded from https://dev.mysql.com/downloads) and found that the symbols in the pdb file DO match the dll file. I used the same dumpbin /PDBPATH:VERBOSE command as the bug reporter to perform the check.

The bug also mentions version 8.0.37 so I've checked the libmysql.dll and libmysql.pdb files from  mysql-8.0.37-winx64.zip and mysql-8.0.37-winx64-debug-test.zip (from MySQL community downloads) as well: these also match.

I suggest that the user uses these zip files to ensure that they have a compatible set of executable and symbol files.

Can't repeat.

All of these are very good news, if it works for you then I had to be wrong somewhere at sure, I check it and let know, thx for verifying it.

Also thx for pointing me to the mysql_client_test.cc example, I know about it ~5 days, there are very good examples at one place, thx

You are truly welcome ......

Ok, I know where is the source of this problem, we are both right 🕺, the problem is that the dll-es and exe-es aren't the same as in the *.zip and in the *.msi installers, now I checked both mysql-9.0.1-winx64 .zip vs .msi and also mysql-8.0.37-winx64 .zip vs .msi.

The PDB files really match if you download eg. mysql-9.0.1-winx64.zip and mysql-9.0.1-winx64-debug-test.zip, but they don't match with the mysql-9.0.1-winx64.msi.

Of course, I've been using .msi installers since forever.

I suppose binaries are signed in .msi installers but PDB files should be DESIGNED to match in these cases (I think, I read it somewhere, not sure about this), which means they must match whether an executable is signed or not.

Are you patching these executables for .msi installers in any non-standard way? May be a very old pathing process not supported by the latest VS2022, but I could be very wrong about this, only guessing, the problem can be anywhere. (I know nothing about signing executables, I've never done it).

I fastly found this https://learn.microsoft.com/en-us/answers/questions/984936/does-signing-a-dll-invalidate-a...

Actually, yes ........

There is also a problem on our side , which is now fixed for future releases ......

There was one technical problem on one of our build machine , which is now fixed. 

Not a bug in the code or packaging or anything similar ......

So, please continue downloading and using the packages that work well .......

Thanks.

And do you think that it can be the cause of PDB mismatch?

The Root Certificate chain isn't correct, I do well with schannel and openssl, only know nothing about specifically exe signing.

This is the output on my dev. machine:

 C:\Program Files\MySQL\MySQL Server 9.0\lib
λ signtool.exe verify /v .\libmysql.dll

Verifying: .\libmysql.dll

Signature Index: 0 (Primary Signature)
Hash of file (sha256): D1825A922C62F1CB4F228BEFA3AF060C7512DE7D1052183E4849BD3807F57941

Signing Certificate Chain:
    Issued to: DigiCert Assured ID Root CA
    Issued by: DigiCert Assured ID Root CA
    Expires:   Mon Nov 10 02:00:00 2031
    SHA1 hash: 0563B8630D62D75ABBC8AB1E4BDFB5A899B24D43

        Issued to: DigiCert Trusted Root G4
        Issued by: DigiCert Assured ID Root CA
        Expires:   Mon Nov 10 01:59:59 2031
        SHA1 hash: A99D5B79E9F1CDA59CDAB6373169D5353F5874C6

            Issued to: DigiCert Trusted G4 Code Signing RSA4096 SHA384 2021 CA1
            Issued by: DigiCert Trusted Root G4
            Expires:   Tue Apr 29 01:59:59 2036
            SHA1 hash: 7B0F360B775F76C94A12CA48445AA2D2A875701C

                Issued to: Oracle America, Inc.
                Issued by: DigiCert Trusted G4 Code Signing RSA4096 SHA384 2021 CA1
                Expires:   Wed Mar 12 01:59:59 2025
                SHA1 hash: CD7BE0F00F2A5EE102C3037E098AF3F457D3B1AB

The signature is timestamped: Sat Jul 13 01:39:14 2024
Timestamp Verified by:
    Issued to: DigiCert Assured ID Root CA
    Issued by: DigiCert Assured ID Root CA
    Expires:   Mon Nov 10 02:00:00 2031
    SHA1 hash: 0563B8630D62D75ABBC8AB1E4BDFB5A899B24D43

        Issued to: DigiCert Trusted Root G4
        Issued by: DigiCert Assured ID Root CA
        Expires:   Mon Nov 10 01:59:59 2031
        SHA1 hash: A99D5B79E9F1CDA59CDAB6373169D5353F5874C6

            Issued to: DigiCert Trusted G4 RSA4096 SHA256 TimeStamping CA
            Issued by: DigiCert Trusted Root G4
            Expires:   Mon Mar 23 01:59:59 2037
            SHA1 hash: B6C8AF834D4E53B673C76872AA8C950C7C54DF5F

                Issued to: DigiCert Timestamp 2023
                Issued by: DigiCert Trusted G4 RSA4096 SHA256 TimeStamping CA
                Expires:   Sat Oct 14 01:59:59 2034
                SHA1 hash: 66F02B32C2C2C90F825DCEAA8AC9C64F199CCF40

SignTool Error: A certificate chain processed, but terminated in a root
        certificate which is not trusted by the trust provider.

Number of files successfully Verified: 0
Number of warnings: 0
Number of errors: 1

Yes, that was the cause.

But, this problem is solved now.

Please, just continue using the package that works .....

This problem will not appear in the next releases of 8.0 , 8.4 and 9.0.

This report is now closed.

Thx for the asap support, I test it when the next GA will be out.

Also the signtool command I posted was wrong it should be: signtool.exe verify /pa /v .\libmysql.dll

Then it correctly verifies the cert. chain, I will not delay, we all have a lot of work to do 😁, thx 👍

Mr. Janovic,

You are truly welcome ......