Bug #51466 Source packages are broken with cmake in a cmake-agnostic environment
Submitted: 24 Feb 2010 16:13 Modified: 28 Feb 2010 21:56
Reporter: Daniel Fischer Email Updates:
Status: Closed Impact on me:
None 
Category:MySQL Server: Packaging Severity:S1 (Critical)
Version: OS:Any
Assigned to: Vladislav Vaintroub CPU Architecture:Any

[24 Feb 2010 16:13] Daniel Fischer 
Description:
When building a source distribution in a cmake-agnostic environment (BUILD/compile-dist && make dist) , and cmake is in PATH, the resulting source package lacks some files that should be included, and includes a large number of files that should not be included.

The following scripts cause bugs by being present in the source package, since they will contain pathnames etc. that were valid during the source build, will not be rebuilt in a subsequent build, and will then fail when run:

+mysql-5.5.99-m3/scripts/make_binary_distribution
+mysql-5.5.99-m3/scripts/msql2mysql
+mysql-5.5.99-m3/scripts/mysql_config
+mysql-5.5.99-m3/scripts/mysql_convert_table_format
+mysql-5.5.99-m3/scripts/mysql_find_rows
+mysql-5.5.99-m3/scripts/mysql_fix_extensions
+mysql-5.5.99-m3/scripts/mysql_install_db
+mysql-5.5.99-m3/scripts/mysql_secure_installation
+mysql-5.5.99-m3/scripts/mysql_setpermission
+mysql-5.5.99-m3/scripts/mysql_zap
+mysql-5.5.99-m3/scripts/mysqlaccess
+mysql-5.5.99-m3/scripts/mysqld_multi
+mysql-5.5.99-m3/scripts/mysqld_safe
+mysql-5.5.99-m3/scripts/mysqldumpslow
+mysql-5.5.99-m3/scripts/mysqlhotcopy

There are some other build-related files whose presence in a source package is a bug, as they must be generated individually for each build:

+mysql-5.5.99-m3/Makefile
+mysql-5.5.99-m3/client/Makefile
+mysql-5.5.99-m3/cmd-line-utils/libedit/Makefile
+mysql-5.5.99-m3/cmd-line-utils/readline/configure.in
+mysql-5.5.99-m3/dbug/Makefile
+mysql-5.5.99-m3/extra/Makefile
+mysql-5.5.99-m3/include/Makefile
+mysql-5.5.99-m3/include/config.h
+mysql-5.5.99-m3/include/my_config.h
+mysql-5.5.99-m3/include/mysql_version.h
+mysql-5.5.99-m3/libmysql/Makefile
+mysql-5.5.99-m3/libmysql/acinclude.m4
+mysql-5.5.99-m3/libmysqld/Makefile
+mysql-5.5.99-m3/libmysqld/examples/Makefile
+mysql-5.5.99-m3/libservices/Makefile
+mysql-5.5.99-m3/man/Makefile
+mysql-5.5.99-m3/mysql-test/Makefile
+mysql-5.5.99-m3/mysys/Makefile
+mysql-5.5.99-m3/plugin/audit_null/Makefile
+mysql-5.5.99-m3/plugin/daemon_example/Makefile
+mysql-5.5.99-m3/plugin/daemon_example/configure.in
+mysql-5.5.99-m3/plugin/fulltext/Makefile
+mysql-5.5.99-m3/plugin/semisync/Makefile
+mysql-5.5.99-m3/plugin/semisync/configure.in
+mysql-5.5.99-m3/regex/Makefile
+mysql-5.5.99-m3/scripts/Makefile
+mysql-5.5.99-m3/sql/Makefile
+mysql-5.5.99-m3/sql/share/Makefile
+mysql-5.5.99-m3/storage/archive/Makefile
+mysql-5.5.99-m3/storage/blackhole/Makefile
+mysql-5.5.99-m3/storage/csv/Makefile
+mysql-5.5.99-m3/storage/example/Makefile
+mysql-5.5.99-m3/storage/federated/Makefile
+mysql-5.5.99-m3/storage/heap/Makefile
+mysql-5.5.99-m3/storage/ibmdb2i/Makefile
+mysql-5.5.99-m3/storage/innobase/Makefile
+mysql-5.5.99-m3/sql-bench/Makefile
+mysql-5.5.99-m3/storage/myisam/Makefile
+mysql-5.5.99-m3/storage/myisammrg/Makefile
+mysql-5.5.99-m3/storage/ndb/src/common/logger/listtest/Makefile
+mysql-5.5.99-m3/storage/ndb/src/common/logger/loggertest/Makefile
+mysql-5.5.99-m3/storage/ndb/src/common/mgmcommon/printConfig/Makefile
+mysql-5.5.99-m3/storage/ndb/src/common/transporter/basictest/Makefile
+mysql-5.5.99-m3/storage/ndb/src/common/transporter/failoverSCI/Makefile
+mysql-5.5.99-m3/storage/ndb/src/common/transporter/perftest/Makefile
+mysql-5.5.99-m3/storage/ndb/src/common/transporter/priotest/Makefile
+mysql-5.5.99-m3/storage/ndb/src/common/transporter/priotest/prioSCI/Makefile
+mysql-5.5.99-m3/storage/ndb/src/common/transporter/priotest/prioSHM/Makefile
+mysql-5.5.99-m3/storage/ndb/src/common/transporter/priotest/prioTCP/Makefile
+mysql-5.5.99-m3/storage/ndb/src/common/util/filetest/Makefile
+mysql-5.5.99-m3/storage/ndb/src/common/util/testConfigValues/Makefile
+mysql-5.5.99-m3/storage/ndb/src/common/util/testProperties/Makefile
+mysql-5.5.99-m3/storage/ndb/src/common/util/testSimpleProperties/Makefile
+mysql-5.5.99-m3/storage/ndb/src/mgmclient/test_cpcd/Makefile
+mysql-5.5.99-m3/storage/ndb/src/mgmapi/test/Makefile
+mysql-5.5.99-m3/storage/ndb/src/ndbapi/signal-sender/Makefile
+mysql-5.5.99-m3/storage/ndb/test/newtonapi/basic_test/Makefile
+mysql-5.5.99-m3/storage/ndb/test/newtonapi/basic_test/basic/Makefile
+mysql-5.5.99-m3/storage/ndb/test/newtonapi/basic_test/bulk_read/Makefile
+mysql-5.5.99-m3/storage/ndb/test/newtonapi/basic_test/ptr_binding/Makefile
+mysql-5.5.99-m3/storage/ndb/test/newtonapi/perf_test/Makefile
+mysql-5.5.99-m3/storage/ndb/test/odbc/SQL99_test/Makefile
+mysql-5.5.99-m3/storage/ndb/test/odbc/client/Makefile
+mysql-5.5.99-m3/storage/ndb/test/odbc/dm-iodbc/Makefile
+mysql-5.5.99-m3/storage/ndb/test/odbc/dm-unixodbc/
+mysql-5.5.99-m3/storage/ndb/test/odbc/dm-unixodbc/Makefile
+mysql-5.5.99-m3/storage/ndb/test/odbc/driver/
+mysql-5.5.99-m3/storage/ndb/test/odbc/driver/Makefile
+mysql-5.5.99-m3/storage/ndb/test/odbc/test_compiler/Makefile
+mysql-5.5.99-m3/storage/perfschema/Makefile
+mysql-5.5.99-m3/storage/perfschema/unittest/Makefile
+mysql-5.5.99-m3/tests/Makefile
+mysql-5.5.99-m3/unittest/mysys/Makefile
+mysql-5.5.99-m3/unittest/mytap/Makefile
+mysql-5.5.99-m3/vio/Makefile

Some files that are missing should not be removed from our packages arbitrarily without a good reason:

Files that are no longer packaged, and should not be removed arbitrarily:

-mysql-5.5.99-m3/client/my_user.c
-mysql-5.5.99-m3/cmd-line-utils/libedit/fcns.c
-mysql-5.5.99-m3/cmd-line-utils/libedit/help.c
-mysql-5.5.99-m3/include/mysqld_error.h
-mysql-5.5.99-m3/libmysql/password.c
-mysql-5.5.99-m3/libmysql_r/conf_to_src.c
-mysql-5.5.99-m3/scripts/mysql_fix_privilege_tables_sql.c
-mysql-5.5.99-m3/sql/share/english/
-mysql-5.5.99-m3/sql/share/english/errmsg.sys
-mysql-5.5.99-m3/storage/ndb/include/ndb_global.h
-mysql-5.5.99-m3/storage/ndb/include/ndb_types.h
-mysql-5.5.99-m3/storage/ndb/include/ndb_version.h
-mysql-5.5.99-m3/storage/ndb/src/common/util/testBitmask.cpp

*cont*

How to repeat:
See above.
[24 Feb 2010 16:14] Daniel Fischer 
More files that should not be added arbitrarily at this stage

Attachment: files.txt (text/plain), 44.79 KiB.
[28 Feb 2010 21:56] Vladislav Vaintroub 
This has been pushed to next-mr-bugfixing as 
Revision: 
2923.2.61 revid:wlad@sun.com-20100224190705-alv4xe1kfetq7rh4

Oddly, I did not use Bug#51466 in the comment, instead just #51466, and post-commit trigger did not work, so it did not make an entry in bugdb. Basically, I introduced some pattern of what not to include, and added most of files mentioned into cmake/cpack_source_ignore_files.cmake

Other examples (file was included prior but not now), show the deficiency of the old system, it packs symlinks and what else :)

Having said this, I do not think it is possible to have an ideal solution using exaclty described environment (not from source repo, in-source build).
What "make dist" with CMake: 
- it tries to get a clean source dir, something like bzr export outside of build directory.

- it produces sql_yacc.{h,cc} and packs it into clean source dir into sql subdirectory

- it run BUILD/autorun.sh in the clean souce directory and removes couple of files that are considered useless or maybe harmful in source distribution such as autom4.te-something.

Where it can get problematic is getting the clean source dir, and both "bzr export" and out-of-source build are  good. In-source build is not so good, the last resort is "make clean" plus custom source exclusion rules as in the patch.

I think with this patch the original problem will go away. However, the source package produced this way might still be not ideal, and there is at least 2 workarounds for it: either running "make dist" in bzr repo or building out-of-source. Admittedly last one might require to make source distribution be a bit more CMake-aware.

As a note, you do not need to run compile-dist script, before running "make dist", "make dist" is self-sufficient.