MySQL Bugs: #52850: mysqld-debug.pdb doesn't match mysqld-debug.exe in 5.5.3 on windows

Bug #52850	mysqld-debug.pdb doesn't match mysqld-debug.exe in 5.5.3 on windows
Submitted:	15 Apr 2010 6:20	Modified:	4 Aug 2010 19:52
Reporter:	Shane Bester (Platinum Quality Contributor)	Email Updates:
Status:	Closed	Impact on me:	None
Category:	MySQL Server: Packaging	Severity:	S1 (Critical)
Version:	mysql-5.5.3-m3-win	OS:	Windows
Assigned to:	Vladislav Vaintroub	CPU Architecture:	Any
Tags:	pdb, regression

Description:
mysqld-debug.exe should correspond to mysqld-debug.pdb in the same directory.
However, mysqld-debug.exe is looking for mysqld.pdb, which means we have to play trickery to get any debugging working.

I checked in debugging tools for windows:

DBGHELP: H:\mysql\5.5\5.5.3\mysql-5.5.3-m3-winx64\bin\mysqld-debug.exe - OK
DBGENG:  H:\mysql\5.5\5.5.3\mysql-5.5.3-m3-winx64\bin\mysqld-debug.exe - Mapped image memory
DBGHELP: H:\mysql\5.5\5.5.3\mysql-5.5.3-m3-winx64\bin\mysqld.pdb - mismatched pdb
DBGHELP: H:\mysql\5.5\5.5.3\mysql-5.5.3-m3-winx64\bin\exe\mysqld.pdb - file not found
DBGHELP: H:\mysql\5.5\5.5.3\mysql-5.5.3-m3-winx64\bin\symbols\exe\mysqld.pdb - file not found
DBGHELP: mysqld.pdb - file not found
DBGHELP: G:\mysql-5.5.3-m3-winbuild\mysql-community-debug-5.5.3-m3-build\sql\Debug\mysqld.pdb - file not found
DBGHELP: Couldn't load mismatched pdb for mysqld-debug.exe
*** ERROR: Symbol file could not be found.  Defaulted to export symbols for mysqld-debug.exe - 

How to repeat:
extract mysql-noinstall-5.5.3-m3-winx64.zip
run mysqld-debug.exe --no-defaults --console --skip-gr --skip-na --core-file

then run any crashing testcase and you'll see
o) wrong/ugly stacktrace printed on console
o) when trying to open the minidump in windbg, try this:

Configure your symbol path to be the bin directory that contains mysqld.debug.pdb then:

!sym noisy
.reload -f mysqld-debug.exe

Suggested fix:
Fix this ASAP, it is a regression packaging bug, hence no risk, and we need proper stack traces from folks in the field.

A workaround is to:

rename mysqld.dbg mysqld.pdb.old
rename mysqld-debug.pdb mysqld.pdb

in case you don't have a crashing testcase handy... BTW, don't try BZR source. use the exact package!

Attachment: bug52850_debug_crasher.sql (application/octet-stream, text), 4.35 KiB.

BTW, don't worry about the crash itself. that could be bug #52159 !

Shane, it does match, if  you rename the mysqld-debug.pdb to mysqld.pdb.
You can debug optimized server. And you can DBUG debug server. Debugging debug server is possible after renaming the PDB. 

Sorry for that, but if you can come up with a proposal how to package both debug and release server into a single package without hacks, I'd love to hear it.

I do not see how it can ever be S1 (it does not affect anyone in production).

vlad, Look at the attached file for full comparisons (using untouched, freshly extracted -no-installs).  This is a real bug and must be refixed by engineering.

mysql-noinstall-5.5.3-m3-winx64.zip <--broken
mysql-noinstall-5.5.3-m3-win32.zip  <--broken
mysql-noinstall-5.5.2-m2-win32.zip  <--works
mysql-advanced-gpl-noinstall-5.1.45-winx64.zip <--works

I crash our server 30 times a day, so this is really S1 for me, the user.

outputs

Attachment: bug52850__proof_of_regression.txt (text/plain), 16.10 KiB.

the build team knows how to do this, they fixed it previously, and now it just broke. maybe the build process is to blame?

I understand it is important for you, but you're just a single and very special user that runs mysqld-debug.exe for anything else than getting DBUG output.

I describe the build process how it worked previously and how it works currently.

When you build from source using cmake and Visual Studio generator, the build will create sql\Debug\mysqld.exe. This is the way how it is normally complied and normally tested on every pushbuild. Tools (mtr etc) always assume mysqld.exe/mysqld.pdb and not mysqld-debug.exe/mysqld-debug.pdb.

mysqld-debug.exe and mysqld-debug.pdb are created by make_win_bin_dist by renaming debug executable and pdb with -debug suffix. Or in the new CMake world they are created (using rename) by

devenv MySQL.sln /build debug
devenv MySQL.sln /build relwithdebinfo /project package

This is how it works without hacks.

Now the hacks of the bad past, which I removed in this release.

Specifically for release, build team had a special parameter that allowed mysqld to have a different output name (mysqld-debug.exe). Consequently, it had a PDB named mysqld-debug.pdb. There was used some post-build processing in special scripts on internals repo to pack the stuff in a way quite different from make_win_bin_dist.

mysqld-debug.exe has a matching mysqld-debug.pdb which is good, everything else is bad.
- first, such build was never tested on the Pushbuild
- second, all debug plugins now have a different dependency (they depend on mysqld-debug.exe, not mysqld-debug.pdb).
- third, such build can never be packed with official tools (which used to be make_win_bin_dist, and now will be, at least for zip, devenv MySQL.sln /build relwithdebinfo /project package)

I good workaround would be to create debug server as mysqld.exe and corresponding PDB as mysqld-debug.pdb. Unfortunately, for Visual Studio generator there is no way to do that currently with CMake, even if VS has the option that corresponds to linker's /PDB switch.

Since this does not work, the only workaround I can offer now is that:
store mysqld.pdb corresponding to mysqld-debug.exe as bin\debug\mysqld.pdb
subdirectory in the zip (and it should be in MSI too, though I do not touch MSI's yet, so I need to leave that to the build team)

I can tweak mysqld.exe's crashhandler that outputs callstack to look for symbols in this directory. When you debug with VS though, you will need to point debugger to the mysqld.pdb in the bin\debug directory.

This is the best thing I can do.

/rant mode
At every possible occasion, I moaned current practice of packing debug and release package together and told that releasing a full debug zip/tgz on every platform is much more cleaner way.

And at any occasion I was told,"..but our support has requested this to be together and it is much more simpler for them to use it as it is now". I do hope one day people will recognize that the old hack breaks everywhere now and should be eliminated.

Attachment: bug52850__proof_of_regression.txt (text/plain), 16483 bytes.
http://bugs.mysql.com/file.php?id=14517

The callstack looks pretty much correct(maybe it is not that bad that we export all symbols from mysqld.exe, after all)

A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/112569

3082 Vladislav Vaintroub	2010-06-30
      Bug #52850: mysqld-debug.pdb doesn't match 
      mysqld-debug.exe in 5.5.3 on windows
      
      Fix:
      
      - Do not rename PDB, install mysqld.pdb matching 
      mysqld-debug.exe into bin\debug subdirectory
      
      - Stack tracing code will now additionally look in 
      debug subdirectory of the application directory 
      for debug symbols.
      
      - Small cleanup in stacktracing code: link with 
      dbghelp rather than load functions dynamically 
      at runtime, since dbghelp.dll is always present.
      
      - Install debug binaries with WiX
     @ cmake/install_macros.cmake
        Add optional COMPONENT and PDB_DESTINATION 
        to INSTALL_DEBUG_TARGET
     @ mysys/stacktrace.c
        If binary is build with DBUG, also look in debug subdirectory
        of  executable directory. Packaging will put some PDBs there
        (e.g bin\mysqld-debug.exe will have corresponding pdb in 
        bin\debug)
        
        Also some cleanup: do not load dbghelp dynamically, instead
        link with it. dbghelp is present on all Windows starting with 
        XP.
     @ packaging/WiX/CPackWixConfig.cmake
        Install debug binaries
     @ sql/CMakeLists.txt
        Do not rename PDB for mysqld-debug.exe, install it in debug subdirectory

queued to trunk-bugfixig, next-mr-bugfixing

i noticed that the following don't have any mysqld-debug.* files now:
mysql-5.5.5-m3-win32.zip
mysql-5.5.5-m3-winx64.zip

Shane, in this case you might want to create a different bug and assign to build team. It is definitely possibile to build and package both debug and release binaries, the question is merely about using this possibility.

A note has been added to the 5.5.6 changelog: 

The mysql-debug.pdb supplied with releases did not match the corresponding mysqld.exe.

Pushed into mysql-trunk 5.5.6-m3 (revid:alik@sun.com-20100723121820-jryu2fuw3pc53q9w) (version source revid:vasil.dimov@oracle.com-20100531152341-x2d4hma644icamh1) (merge vers: 5.5.5-m3) (pib:18)

Pushed into mysql-next-mr (revid:alik@sun.com-20100723121929-90e9zemk3jkr2ocy) (version source revid:vasil.dimov@oracle.com-20100531152341-x2d4hma644icamh1) (pib:18)

Already noted in changelog

Pushed into mysql-trunk 5.6.1-m4 (revid:alik@ibmvm-20100804080001-bny5271e65xo34ig) (version source revid:vasil.dimov@oracle.com-20100531152341-x2d4hma644icamh1) (merge vers: 5.5.5-m3) (pib:18)

Pushed into mysql-trunk 5.6.1-m4 (revid:alik@ibmvm-20100804081533-c1d3rbipo9e8rt1s) (version source revid:vasil.dimov@oracle.com-20100531152341-x2d4hma644icamh1) (merge vers: 5.5.5-m3) (pib:18)

Bug is not present in any released 5.6.x version.