MySQL Bugs: #115734: Build is so "fast" and uses so many jobs that it crashes the system

Bug #115734	Build is so "fast" and uses so many jobs that it crashes the system
Submitted:	1 Aug 2024 4:54	Modified:	19 Jun 13:26
Reporter:	Eli Schwartz	Email Updates:
Status:	Verified	Impact on me:	None
Category:	Connector / C++	Severity:	S3 (Non-critical)
Version:	8.0.33	OS:	Any
Assigned to:		CPU Architecture:	Any

Description:
I tried to build mysql-connector-c++ on Gentoo Linux using the following command:

```
emerge mysql-connector-c++
```

It configured the sources with the following command: cmake -C /var/tmp/portage/dev-db/mysql-connector-c++-8.0.33-r1/work/mysql-connector-c++-8.0.33-src_build/gentoo_common_config.cmake -G Unix Makefiles -DCMAKE_INSTALL_PREFIX=/usr -DBUNDLE_DEPENDENCIES=OFF -DWITH_LZ4=system -DWITH_SSL=system -DWITH_ZLIB=system -DWITH_ZSTD=system -DWITH_JDBC=yes -DMYSQLCLIENT_STATIC_BINDING=0 -DMYSQLCLIENT_STATIC_LINKING=0 -DCMAKE_BUILD_TYPE=RelWithDebInfo -DCMAKE_TOOLCHAIN_FILE=/var/tmp/portage/dev-db/mysql-connector-c++-8.0.33-r1/work/mysql-connector-c++-8.0.33-src_build/gentoo_toolchain.cmake /var/tmp/portage/dev-db/mysql-connector-c++-8.0.33-r1/work/mysql-connector-c++-8.0.33-src

And started compiling. Shortly thereafter, my system froze.

Through careful timing, I discovered by watching the process monitor at the same time I started trying to build mysql-connector-c++, that running "make" actually spawns a new copy of cmake, using the arguments:

--build . --parallel

If I let it keep running after that, my system crashes. If I CTRL+C immediately, my system is saved.

Looking through the source code, I see the bug here: https://github.com/mysql/mysql-connector-cpp/blob/ac218f0d2d5b7957a8a2efee148bc8d7a1c623a5...

Please do NOT do this. It is not "faster" to use parallel builds when running a recursive build system. You end up running as many jobs as you have cores *times* the number of dependencies, plus one more for mysql-connector-c++ itself.

This ends up being several (and perhaps many) times the number of parallel cores you have available.

How to repeat:
emerge mysql-connector-c++

Suggested fix:
Simply do not do this. It is dangerous.

Ideally, use cmake's embed mode to allow cmake to build the entire project as well as its dependencies in the same Make build system. This will allow users to specify their own -j value to mysql-connector-c++ itself, and have GNU Make parallelize across all dependencies at the same time, while tracking job availability without over-counting. The user is probably already passing -j$(nproc), and all available build edges for the parent project as well as bundled dependencies would *collaborate* together, to consume exactly $(nproc) number of build jobs in total.

Hello Eli Schwartz,

Thank you for the bug report.
Could you please try with latest version and report us back if issue persist even in latest version along with details? Thank you.

Regards,
Ashwini Patil

No feedback was provided for this bug for over a month, so it is
being suspended automatically. If you are able to provide the
information that was originally requested, please do so and change
the status of the bug back to "Open".

I have tested 9.3.0 and it is still a problem.

Hello Eli Schwartz,

Thank you for the feedback.
Verified as described.

Regards,
Ashwini Patil

Posted by developer:
 
I can agree that "blindly" adding --parallel option as we do might not be a good thing in all situations. I think we will follow your advice and remove that "optimization" -- people for whom it is too slow can use ninja generator as a faster alternative.