MySQL Bugs: #116624: [MSVC] Race condition occurs during msbuild with VS2022

Bug #116624	[MSVC] Race condition occurs during msbuild with VS2022
Submitted:	12 Nov 2024 6:58	Modified:	27 Nov 2024 8:12
Reporter:	Sung Hui Chua	Email Updates:
Status:	Verified	Impact on me:	None
Category:	MySQL Server: Compiling	Severity:	S3 (Non-critical)
Version:	latest commit	OS:	Windows (Windows 11)
Assigned to:		CPU Architecture:	x86
Tags:	build error, build from source, msvc

Description:
Hi, I work on MSVC compiler testing, and we regularly build popular open-source project including MySQL, with development builds of MSVC and tested with various cl options and different target architectures in order to find and fix regressions before they ship and cause problems for you.

As I tried to build MySQL at branch 8.4 with MSVC, I encountered the errors as below:

```
  1194>CustomBuild:
         Running C++ protobuf compiler on C:/gitP/mysql/mysql-server/plugin/x/protocol/protobuf/mysqlx_prepare.proto 
  1198>ClCompile:
         mysqlx_prepare.pb.cc
  1194>CustomBuild:
         C:/gitP/mysql/mysql-server/build_amd64/plugin/x/generated/protobuf/mysqlx_prepare.pb.cc: Permission denied
  1194>C:\Program Files\Microsoft Visual Studio\2022\Enterprise\MSBuild\Microsoft\VC\v170\Microsoft.CppCommon.targets(254,5): error MSB8066: Custom build for 'C:\gitP\mysql\mysql-server\build_amd64\CMakeFiles\697f4a8d36a4375264217bb7617984f7\mysqlx_connection.pb.cc.rule;C:\gitP\mysql\mysql-server\build_amd64\CMakeFiles\697f4a8d36a4375264217bb7617984f7\mysqlx_crud.pb.cc.rule;C:\gitP\mysql\mysql-server\build_amd64\CMakeFiles\697f4a8d36a4375264217bb7617984f7\mysqlx_cursor.pb.cc.rule;C:\gitP\mysql\mysql-server\build_amd64\CMakeFiles\697f4a8d36a4375264217bb7617984f7\mysqlx_datatypes.pb.cc.rule;C:\gitP\mysql\mysql-server\build_amd64\CMakeFiles\697f4a8d36a4375264217bb7617984f7\mysqlx_expect.pb.cc.rule;C:\gitP\mysql\mysql-server\build_amd64\CMakeFiles\697f4a8d36a4375264217bb7617984f7\mysqlx_expr.pb.cc.rule;C:\gitP\mysql\mysql-server\build_amd64\CMakeFiles\697f4a8d36a4375264217bb7617984f7\mysqlx_notice.pb.cc.rule;C:\gitP\mysql\mysql-server\build_amd64\CMakeFiles\697f4a8d36a4375264217bb7617984f7\mysqlx_prepare.pb.cc.rule;C:\gitP\mysql\mysql-server\build_amd64\CMakeFiles\697f4a8d36a4375264217bb7617984f7\mysqlx.pb.cc.rule;C:\gitP\mysql\mysql-server\build_amd64\CMakeFiles\697f4a8d36a4375264217bb7617984f7\mysqlx_resultset.pb.cc.rule;C:\gitP\mysql\mysql-server\build_amd64\CMakeFiles\697f4a8d36a4375264217bb7617984f7\mysqlx_session.pb.cc.rule;C:\gitP\mysql\mysql-server\build_amd64\CMakeFiles\697f4a8d36a4375264217bb7617984f7\mysqlx_sql.pb.cc.rule;C:\gitP\mysql\mysql-server\plugin\x\protocol\protobuf\CMakeLists.txt' exited with code 1. [C:\gitP\mysql\mysql-server\build_amd64\plugin\x\protocol\protobuf\mysqlxmessages_HEADERS.vcxproj]
  1194>Done Building Project "C:\gitP\mysql\mysql-server\build_amd64\plugin\x\protocol\protobuf\mysqlxmessages_HEADERS.vcxproj" (Rebuild target(s)) -- FAILED.
```

The race condition occurs during MSBuild with Visual Studio 2022 because two threads are simultaneously attempting to access and modify the same file. Specifically, one thread (node 1198) is compiling <mysqlx_prepare.pb.cc>, while another thread (node 1194) is trying to regenerate the same file using protoc.exe, leading to a permission denied error.

Note: This issue occurs about 8-9 times out of 10 builds.

How to repeat:
1. git clone https://github.com/mysql/mysql-server.git
2. cd mysql/mysql-server
3. mkdir build_amd64
3. cd build_amd64
4. cmake -G "Visual Studio 17 2022" -A x64 -DCMAKE_SYSTEM_VERSION=10.0.22621.0 -DDOWNLOAD_BOOST=1 -DWITH_BOOST=C:\gitP\mysql\mysql-server\boost_1_77 -DWITH_SSL=C:\gitP\Microsoft\vcpkg\installed\x64-windows ..
5. msbuild /m /p:Platform=x64 /p:Configuration=Release MySQL.sln /t:Rebuild

Suggested fix:
Consider adding a check to verify the existence of file before running protoc.exe to generate the files?

Hello Sung Hui Chua,

Thank you for the report and feedback.
I quickly tried to reproduce the issue at my end but not seeing any issues. Am I missing anything here? Please let us know. Thank you.

-- 5/5 times seen this

-- Env

D:\ushastry\builds\mysql-8.4.3\build_amd64>systeminfo

Host Name:                 *******
OS Name:                   Microsoft Windows Server 2022 Standard
OS Version:                10.0.20348 N/A Build 20348
OS Manufacturer:           Microsoft Corporation
OS Configuration:          Member Server
OS Build Type:             Multiprocessor Free
Registered Owner:          Administrator
Registered Organization:   Oracle USA
Product ID:                *******
Original Install Date:     1/25/2022, 6:09:01 PM
System Boot Time:          9/30/2024, 8:42:08 AM
System Manufacturer:       QEMU
System Model:              Standard PC (i440FX + PIIX, 1996)
System Type:               x64-based PC
Processor(s):              1 Processor(s) Installed.
                           [01]: AMD64 Family 25 Model 1 Stepping 1 AuthenticAMD ~2445 Mhz

-- 8.4.3 (-DWITH_BOOST & _DDOWNLOAD_BOOST ignored)

1. Downloaded 8.4.3 source build from internal repository, extracted to D:\ushastry\builds\mysql-8.4.3
2. Opened VS2022 x64 Native Tools command.
3. mkdir D:\ushastry\builds\mysql-8.4.3\build_amd64 & cd /d D:\ushastry\builds\mysql-8.4.3\build_amd64
4. D:\ushastry\builds\cmake-3.31.0\bin\cmake -G "Visual Studio 17 2022" -A x64 -DCMAKE_SYSTEM_VERSION=10.0.22621.0 -DDOWNLOAD_BOOST=1 -DWITH_BOOST=D:\ushastry\builds\mysql-8.4.3\boost_1_77 -DWITH_SSL=C:\OpenSSL-Win64 ..
5. msbuild /m /p:Platform=x64 /p:Configuration=Release MySQL.sln /t:Rebuild
.
.
Build succeeded.

       "D:\ushastry\builds\mysql-8.4.3\build_amd64\MySQL.sln" (Rebuild target) (1) ->
       "D:\ushastry\builds\mysql-8.4.3\build_amd64\ALL_BUILD.vcxproj.metaproj" (Rebuild target) (2) ->
       "D:\ushastry\builds\mysql-8.4.3\build_amd64\testclients\bug25714.vcxproj.metaproj" (Rebuild target) (156) ->
       "D:\ushastry\builds\mysql-8.4.3\build_amd64\libmysql\mysqlclient.vcxproj.metaproj" (Rebuild target) (157) ->
       "D:\ushastry\builds\mysql-8.4.3\build_amd64\libmysql\mysqlclient.vcxproj" (Rebuild target) (962) ->
       (Lib target) ->
         libcrypto.lib(libcrypto-3-x64.dll) : warning LNK4006: __NULL_IMPORT_DESCRIPTOR already defined in libssl.lib(libssl-3-x64.dll); second definition ignored [D:\ushastry\builds\mysql-8.4.3\build_amd64\libmysql\mysqlclient.vcxproj]

    1 Warning(s)
    0 Error(s)

Time Elapsed 00:39:31.23

regards,
Umesh

Hi Umesh,

In fact, the time it takes for your machine to build MySQL is significantly shorter than on mine. This has led me to suspect there might be an issue with my machine. On average, my machine takes at least 1.5 hours to build MySQL. I even set up a new virtual machine to build MySQL, but it is still encountering the same issue.

Here is the VM spec:
OS Name:                   Microsoft Windows Server 2022 Datacenter
OS Version:                10.0.20348 N/A Build 20348
OS Manufacturer:           Microsoft Corporation
OS Configuration:          Standalone Server
OS Build Type:             Multiprocessor Free
Registered Owner:          N/A
Registered Organization:   N/A
Product ID:                **
Original Install Date:     11/14/2024, 6:21:53 AM
System Boot Time:          11/14/2024, 6:26:33 AM
System Manufacturer:       Microsoft Corporation
System Model:              Virtual Machine
System Type:               x64-based PC
Processor(s):              1 Processor(s) Installed.
                           [01]: AMD64 Family 25 Model 1 Stepping 1 AuthenticAMD ~2445 Mhz

Hello Sung Hui Chua,

Thank you for the details.
I'll check if there is any spare box to try this once again and let you know how it goes.
Please expect a delay in response. Thank you.

Sincerely,
Umesh

Hi Umesh,

Sounds good to me. Thanks a lot!

I cannot reproduce either, but that failing library shouldn't be built at all.
Does this patch help?

diff --git a/cmake/protobuf_proto_compile.cmake b/cmake/protobuf_proto_compile.cmake
index e61920aa8ea5..9a8ea26eb81d 100644
--- a/cmake/protobuf_proto_compile.cmake
+++ b/cmake/protobuf_proto_compile.cmake
@@ -217,6 +217,9 @@ FUNCTION(MYSQL_PROTOBUF_GENERATE_CPP_LIBRARY TARGET_NAME)
     # An interface library target may be created with source files.
     ADD_LIBRARY(${TARGET_NAME}_HEADERS INTERFACE ${PROTO_HDRS})
     SET_PROPERTY(TARGET ${TARGET_NAME}_HEADERS PROPERTY EXCLUDE_FROM_ALL TRUE)
+    IF(WIN32)
+      SET_PROPERTY(TARGET ${TARGET_NAME} PROPERTY EXCLUDE_FROM_DEFAULT_BUILD TRUE)
+    ENDIF()
     ADD_DEPENDENCIES(protobuf_generated_all ${TARGET_NAME}_HEADERS)
   ELSE()