| Bug #61340 | execute binary tools from local dir | ||
|---|---|---|---|
| Submitted: | 28 May 2011 14:31 | Modified: | 13 Sep 2018 14:51 |
| Reporter: | Heiko Zuerker | Email Updates: | |
| Status: | Not a Bug | Impact on me: | |
| Category: | MySQL Server: Compiling | Severity: | S3 (Non-critical) |
| Version: | 5.5.11 | OS: | Any |
| Assigned to: | CPU Architecture: | Any | |
| Tags: | comp_err, comp_sql, gen_lex_hash, path | ||
[28 May 2011 17:52]
Vladislav Vaintroub
Heiko, did you found the place in the build that would set the PATH to the directories containing the tools? Hint: there is no such place.. This works because CMake s smart enough to recognize tools it builds itself inside the same build. thus CMake calls those tools with full path. run make VERBOSE=1 to see it.
[28 May 2011 21:11]
Heiko Zuerker
Vladislav, It definitely doesn't find the tools with either adding ./ to the path or applying the patch I attached. Maybe it has something to do with the cross-compiling? I'm using buildroot. Since I'm cross compiling, I first compile the tools with the host gcc and then copy them into the source tree so the subsequent compile uses them. Heiko ------------------------------ [ 0%] Generating ../include/mysqld_error.h, ../sql/share/english/errmsg.sys make[3]: Leaving directory `/data/devil-linux/output/build/mysql-5.5.12' /data/devil-linux/output/host/usr/bin/cmake -E cmake_progress_report /data/devil-linux/output/build/mysql-5.5.12/CMakeFiles cd /data/devil-linux/output/build/mysql-5.5.12/extra && comp_err --charset=/data/devil-linux/output/build/mysql-5.5.12/sql/share/charsets --out-dir=/data/devil-linux/output/build/mysql-5.5.12/sql/share/ --header_file=/data/devil-linux/output/build/mysql-5.5.12/include/mysqld_error.h --name_file=/data/devil-linux/output/build/mysql-5.5.12/include/mysqld_ername.h --state_file=/data/devil-linux/output/build/mysql-5.5.12/include/sql_state.h --in_file=/data/devil-linux/output/build/mysql-5.5.12/sql/share/errmsg-utf8.txt [ 0%] Built target INFO_SRC /bin/sh: comp_err: command not found /usr/bin/make -f strings/CMakeFiles/strings.dir/build.make strings/CMakeFiles/strings.dir/depend make[3]: *** [include/mysqld_error.h] Error 127 make[3]: Leaving directory `/data/devil-linux/output/build/mysql-5.5.12' make[3]: Leaving directory `/data/devil-linux/output/build/mysql-5.5.12' /data/devil-linux/output/host/usr/bin/cmake -E cmake_progress_report /data/devil-linux/output/build/mysql-5.5.12/CMakeFiles make[2]: *** [extra/CMakeFiles/GenError.dir/all] Error 2 make[2]: *** Waiting for unfinished jobs.... [ 0%] Built target INFO_BIN
[29 May 2011 10:01]
Davi Arnaut
comp_err does not get built when cross-compiling. File extra/CMakeLists.txt: IF(NOT CMAKE_CROSSCOMPILING) ADD_EXECUTABLE(comp_err comp_err.c) TARGET_LINK_LIBRARIES(comp_err mysys) ENDIF() Yet the command is executed…
[29 May 2011 13:51]
Heiko Zuerker
Hey, the file info_macros.cmake has set: SET(CMAKE_CROSSCOMPILING "TRUE") Without first compiling the tools under the host gcc and setting the file date into the future, cross-compiling never works since it always wants to execute these binaries. I did a lot of research and it seems that's exactly what everybody else is doing in order to get the cross-compiling to work. But this issue is not about the cross-compiling, it's about finding the binaries. (Maybe it's related) Thanks Heiko
[29 May 2011 14:11]
Davi Arnaut
So, is comp_err actually in the build directory? or somewhere else? You said that it does not work even when using dot-slash. I'm guessing comp_err wasn't built at all.
[29 May 2011 14:20]
Heiko Zuerker
Hi,
Let me back off a little bit, so you understand the cross-compile build process we have to go through:
1) compile parts of mysql-5.5.x on the host with the host gcc (non-cross-compile):
make -C $(@D)/vio
make -C $(@D)/mysys
make -C $(@D)/strings
make -C $(@D)/dbug
make -C $(@D)/regex
make -C $(@D)/sql gen_lex_hash
make -C $(@D)/scripts comp_sql
make -C $(@D)/extra comp_err
You need to do this step, otherwise the tools will get compiled for the target and the binaries between the machine you compile on and the machine you'll later run the binaries on are of course not compatible. This means the compile will otherwise fail when it tries to execute e.g. sql gen_lex_hash.
2) unpack the sources again and copy sql/gen_lex_hash,scripts/comp_sql,extra/comp_err from before into the source tree. Change the file dates into the future so the tools won't get recompiled and break (since they would be for the target architecture)
3) cross-compile mysql for the target.
This whole process works like a charm with the exception that the binary tools gen_lex_hash, comp_sql and comp_err can not be found and the compile breaks. The fix for this is either adding ./ to the PATH or applying the patch I initially submitted (which just adds a ./ in front of the tools before they get executed). Adding ./ to the PATH upsets a lot of people, since they consider it a security risk, thus I created the patch.
Heiko
[29 May 2011 17:06]
Davi Arnaut
ADD_CUSTOM_COMMAND documentation states:
"If COMMAND specifies an executable target (created by ADD_EXECUTABLE) it will automatically be replaced by the location of the executable created at build time. Additionally a target-level dependency will be added so that the executable target will be built before any target using this custom command."
The problem is that when cross-compiling comp_err (for example) is not an executable target created by ADD_EXECUTABLE. Hence, cmake won't automatically replace the location with the full path, causing problems when cross-compiling.
A problem with the suggested dot-slash patch is that cmake won't create a target-level dependency, breaking non-cross-compiling builds. In order for this to work, something like ${PROJECT_BINARY_DIR} should be used instead of dot-slash and the dependency needs to be wired manually.
[29 May 2011 18:11]
Vladislav Vaintroub
The cross-compiling story for MySQL is not yet a good one. I managed to have a MinGW on Linux build some time (~1.5 years) ago and it in fact required comp_err, gen_lex_hash and comp_sql "native"(e.g Linux) executables on the build machine in the PATH. So in that case you are right, cross-compiling would in fact require tools in the PATH. But this case is of very little applicability to MySQL yet (it is not even documented how cross-compile should work, and I do not think development team is interested taking care that cross-compiling). Anyway, having ./ prefix would not solve cross-compiling - it is much more practical to have build tools in one place which is in PATH rather than copy them to different expected places in the build directory. Maybe, cross-compilation that is done right way should do FIND_PATH() for those tools, I dunno.
[29 May 2011 21:13]
Vladislav Vaintroub
http://www.vtk.org/Wiki/CMake_Cross_Compiling describes how to handle this situation cleanly, in section "Using executables in the build created during the build" The technique uses EXPORT for tools during native build and IMPORT during cross-build.
[13 Sep 2018 14:51]
Dyre Tjeldvoll
Posted by developer: I'm closing this as not a bug since invoking built binaries from works as intended, and does not depend on having current directory in path. The issue of cross-compiling is a larger one which would require its own WL.
[2 Sep 2020 14:23]
Evgeny Dontsov
version: mysql-connector-c 6.1.11
os: Debian 9.9, Ubuntu 16.04, Ubuntu 18.04, Oracle Linux 7.7
We catch this error when use custom path gcc and our toolchain.cmake.
"
SET(CMAKE_SYSTEM_NAME Linux)
SET(CMAKE_C_COMPILER "${gccDir}/bin/gcc" )
SET(CMAKE_CXX_COMPILER "${gccDir}/bin/g++" )
SET(CMAKE_LIBRARY_PATH "${PRJ_ENV}/lib" "${gccDir}/lib" "${gccDir}/lib64")
SET(CMAKE_INCLUDE_PATH "${PRJ_ENV}/include" "${gccDir}/include")
SET(CMAKE_PREFIX_PATH "${PRJ_ENV}" "${gccDir}")
set(CMAKE_FIND_ROOT_PATH "${PRJ_ENV}" "${gccDir}" )
set(CMAKE_FIND_ROOT_PATH_MODE_PROGRAM BOTH)
set(CMAKE_FIND_ROOT_PATH_MODE_LIBRARY ONLY)
set(CMAKE_FIND_ROOT_PATH_MODE_INCLUDE ONLY)
"
CMake failed with "comp_err: not found" message:
[16:35:54]: [Step 4/4] [ 89%] Built target comp_err
[16:35:54]: [Step 4/4] Scanning dependencies of target GenError
[16:35:54]: [Step 4/4] [ 89%] Generating ../include/mysqld_error.h, ../sql/share/english/errmsg.sys
[16:35:54]: [Step 4/4] /bin/sh: 1: comp_err: not found
[16:35:54]: [Step 4/4] make[2]: *** [include/mysqld_error.h] Error 127
[16:35:54]: [Step 4/4] make[1]: *** [extra/CMakeFiles/GenError.dir/all] Error 2
[16:35:54]: [Step 4/4] extra/CMakeFiles/GenError.dir/build.make:63: recipe for target 'include/mysqld_error.h' failed
[16:35:54]: [Step 4/4] CMakeFiles/Makefile2:1037: recipe for target 'extra/CMakeFiles/GenError.dir/all' failed
[16:35:54]: [Step 4/4] Makefile:149: recipe for target 'all' failed
[16:35:54]: [Step 4/4] make: *** [all] Error 2
We can't understand the reason. Try to add /extra to PATH - failed.
We call cmake with a lot of arguments:
'CMakeLists.txt', '-DWITH_DEBUG=1', '-DWITHOUT_SERVER=1', '-DHAVE_READDIR_R=0', '-DMYSQL_MAINTAINER_MODE=0', '-DCMAKE_TOOLCHAIN_FILE=toolchain.cmake', '-DSTACK_DIRECTION=-1', '-DHAVE_CXX_FLOATING_POINT_OPTIMIZATION_PROBLEMS_EXITCODE=1', '-DHAVE_C_FLOATING_POINT_OPTIMIZATION_PROBLEMS_EXITCODE=1', '-DHAVE_LLVM_LIBCPP_EXITCODE=1', '-DCMAKE_CROSSCOMPILING=0'
The problem solved with this configure.cmake patch:
IF(${CMAKE_SYSTEM_PROCESSOR})
STRING(TOLOWER ${CMAKE_SYSTEM_PROCESSOR} processor)
ELSE()
STRING(TOLOWER ${CMAKE_HOST_SYSTEM_PROCESSOR} processor)
ENDIF()
Also:
1. mysql_client_test doesn't use CMAKE_TOOLCHAIN_FILE
2. We got some warning that fail build, so we add CMAKE_MAINTAINER_MODE=0
Description: During compilation the tools comp_sql,comp_err,gen_lex_hash are executed without a path prefix in front of them. This means ./ has to be in the path, but this is considered a security risk by some people. The attached patch adds a ./ as path, ensuring the binary tools are found. How to repeat: Compile mysql server without ./ in the path Suggested fix: diff -ruN mysql-5.5.11.orig/extra/CMakeLists.txt mysql-5.5.11/extra/CMakeLists.txt --- mysql-5.5.11.orig/extra/CMakeLists.txt 2011-03-31 08:36:18.000000000 -0500 +++ mysql-5.5.11/extra/CMakeLists.txt 2011-05-14 20:17:05.000000000 -0500 @@ -33,7 +33,7 @@ ADD_CUSTOM_COMMAND(OUTPUT ${PROJECT_BINARY_DIR}/include/mysqld_error.h ${PROJECT_BINARY_DIR}/sql/share/english/errmsg.sys - COMMAND comp_err + COMMAND ./comp_err --charset=${PROJECT_SOURCE_DIR}/sql/share/charsets --out-dir=${PROJECT_BINARY_DIR}/sql/share/ --header_file=${PROJECT_BINARY_DIR}/include/mysqld_error.h diff -ruN mysql-5.5.11.orig/scripts/CMakeLists.txt mysql-5.5.11/scripts/CMakeLists.txt --- mysql-5.5.11.orig/scripts/CMakeLists.txt 2011-03-31 08:36:18.000000000 -0500 +++ mysql-5.5.11/scripts/CMakeLists.txt 2011-05-14 20:18:12.000000000 -0500 @@ -47,7 +47,7 @@ ADD_CUSTOM_COMMAND( OUTPUT ${CMAKE_CURRENT_BINARY_DIR}/mysql_fix_privilege_tables_sql.c ${CAT_COMMAND} - COMMAND comp_sql + COMMAND ./comp_sql mysql_fix_privilege_tables mysql_fix_privilege_tables.sql mysql_fix_privilege_tables_sql.c diff -ruN mysql-5.5.11.orig/sql/CMakeLists.txt mysql-5.5.11/sql/CMakeLists.txt --- mysql-5.5.11.orig/sql/CMakeLists.txt 2011-03-31 08:36:18.000000000 -0500 +++ mysql-5.5.11/sql/CMakeLists.txt 2011-05-14 20:18:58.000000000 -0500 @@ -189,7 +189,7 @@ ADD_CUSTOM_COMMAND( OUTPUT ${CMAKE_CURRENT_BINARY_DIR}/lex_hash.h - COMMAND gen_lex_hash > lex_hash.h + COMMAND ./gen_lex_hash > lex_hash.h DEPENDS gen_lex_hash )