Bug #42969 Please add a MANIFEST to each build
Submitted: 18 Feb 2009 13:33 Modified: 14 Apr 2011 13:45
Reporter: Philip Stoev Email Updates:
Status: Closed Impact on me:
None 
Category:MySQL Server: Packaging Severity:S3 (Non-critical)
Version:5.1,6.0 OS:Any
Assigned to: Joerg Bruehe CPU Architecture:Any

[18 Feb 2009 13:33] Philip Stoev
Description:
It would be really nice to have a MANIFEST file in each build (regardless of the method the build was produced) describing giving information on how the binary was built, similar to perl -V , however without actually hacking mysqld (a separate bug would be filed for that).

Currently, mysqlbug does provide some information, however it is sometimes confusing, for example because both the GCC version that produced mysqld and the GCC version installed on the local machine are given (without indication which is which).

How to repeat:
Download a mysqld package from PB2 and then try to figure where it came from and how it was compiled...

Suggested fix:
Below is an example of the output provided by perl -V . In addition to that, we would need the output provided by bzr version-info and the like.

Summary of my perl5 (revision 5 version 10 subversion 0) configuration:
  Platform:
    osname=linux, osvers=2.6.18-92.1.10.el5, archname=x86_64-linux-thread-multi
    uname='linux x86-7.fedora.phx.redhat.com 2.6.18-92.1.10.el5 #1 smp wed jul 2                                                                              3 03:56:11 edt 2008 x86_64 x86_64 x86_64 gnulinux '
    config_args='-des -Doptimize=-O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fex                                                                              ceptions -fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic -DPERL_                                                                              USE_SAFE_PUTENV -Dversion=5.10.0 -Dmyhostname=localhost -Dperladmin=root@localho                                                                              st -Dcc=gcc -std=gnu99 -Dcf_by=Red Hat, Inc. -Dprefix=/usr -Dvendorprefix=/usr -                                                                              Dsiteprefix=/usr/local -Dprivlib=/usr/lib/perl5/5.10.0 -Dsitelib=/usr/local/lib/                                                                              perl5/site_perl/5.10.0 -Dvendorlib=/usr/lib/perl5/vendor_perl/5.10.0 -Darchlib=/                                                                              usr/lib64/perl5/5.10.0/x86_64-linux-thread-multi -Dsitearch=/usr/local/lib64/per                                                                              l5/site_perl/5.10.0/x86_64-linux-thread-multi -Dvendorarch=/usr/lib64/perl5/vend                                                                              or_perl/5.10.0/x86_64-linux-thread-multi -Dinc_version_list=none -Darchname=x86_                                                                              64-linux-thread-multi -Dlibpth=/usr/local/lib64 /lib64 /usr/lib64 -Duseshrplib -                                                                              Dusethreads -Duseithreads -Duselargefiles -Dd_dosuid -Dd_semctl_semun -Di_db -Ui                                                                              _ndbm -Di_gdbm -Di_shadow -Di_syslog -Dman3ext=3pm -Duseperlio -Dinstallusrbinpe                                                                              rl=n -Ubincompat5005 -Uversiononly -Dpager=/usr/bin/less -isr -Dd_gethostent_r_p                                                                              roto -Ud_endhostent_r_proto -Ud_sethostent_r_proto -Ud_endprotoent_r_proto -Ud_s                                                                              etprotoent_r_proto -Ud_endservent_r_proto -Ud_setservent_r_proto -Dscriptdir=/us                                                                              r/bin -Dotherlibdirs=/usr/local/lib/perl5/site_perl:/usr/local/lib64/perl5/site_                                                                              perl'
    hint=recommended, useposix=true, d_sigaction=define
    useithreads=define, usemultiplicity=define
    useperlio=define, d_sfio=undef, uselargefiles=define, usesocks=undef
    use64bitint=define, use64bitall=define, uselongdouble=undef
    usemymalloc=n, bincompat5005=undef
  Compiler:
    cc='gcc -std=gnu99', ccflags ='-D_REENTRANT -D_GNU_SOURCE -DDEBUGGING -fno-s                                                                              trict-aliasing -pipe -I/usr/local/include -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BIT                                                                              S=64 -I/usr/include/gdbm',
    optimize='-O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-pr                                                                              otector --param=ssp-buffer-size=4 -m64 -mtune=generic -DPERL_USE_SAFE_PUTENV',
    cppflags='-D_REENTRANT -D_GNU_SOURCE -DDEBUGGING -fno-strict-aliasing -pipe                                                                               -I/usr/local/include -I/usr/include/gdbm'
    ccversion='', gccversion='4.3.2 20081105 (Red Hat 4.3.2-7)', gccosandvers=''
    intsize=4, longsize=8, ptrsize=8, doublesize=8, byteorder=12345678
    d_longlong=define, longlongsize=8, d_longdbl=define, longdblsize=16
    ivtype='long', ivsize=8, nvtype='double', nvsize=8, Off_t='off_t', lseeksize                                                                              =8
    alignbytes=8, prototype=define
  Linker and Libraries:
    ld='gcc -std=gnu99', ldflags =''
    libpth=/usr/local/lib64 /lib64 /usr/lib64
    libs=-lresolv -lnsl -lgdbm -ldb -ldl -lm -lcrypt -lutil -lpthread -lc
    perllibs=-lresolv -lnsl -ldl -lm -lcrypt -lutil -lpthread -lc
    libc=, so=so, useshrplib=true, libperl=libperl.so
    gnulibc_version='2.9'
  Dynamic Linking:
    dlsrc=dl_dlopen.xs, dlext=so, d_dlsymun=undef, ccdlflags='-Wl,-E -Wl,-rpath,                                                                              /usr/lib64/perl5/5.10.0/x86_64-linux-thread-multi/CORE'
    cccdlflags='-fPIC', lddlflags='-shared -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOU                                                                              RCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=gener                                                                              ic -DPERL_USE_SAFE_PUTENV'

Characteristics of this binary (from libperl):
  Compile-time options: DEBUGGING MULTIPLICITY PERL_DONT_CREATE_GVSV
                        PERL_IMPLICIT_CONTEXT PERL_MALLOC_WRAP
                        PERL_TRACK_MEMPOOL PERL_USE_SAFE_PUTENV
                        USE_64_BIT_ALL USE_64_BIT_INT USE_ITHREADS
                        USE_LARGE_FILES USE_PERLIO USE_REENTRANT_API
  Built under linux
  Compiled at Nov 28 2008 19:15:11
  @INC:
    /usr/lib64/perl5/5.10.0/x86_64-linux-thread-multi
    /usr/lib/perl5/5.10.0
    /usr/local/lib64/perl5/site_perl/5.10.0/x86_64-linux-thread-multi
    /usr/local/lib/perl5/site_perl/5.10.0
    /usr/lib64/perl5/vendor_perl/5.10.0/x86_64-linux-thread-multi
    /usr/lib/perl5/vendor_perl/5.10.0
    /usr/lib/perl5/vendor_perl
    /usr/local/lib/perl5/site_perl/5.10.0
    /usr/local/lib/perl5/site_perl
    /usr/local/lib64/perl5/site_perl/5.10.0/x86_64-linux-thread-multi
    /usr/local/lib64/perl5/site_perl/5.10.0
    /usr/local/lib64/perl5/site_perl
    .
[31 Aug 2010 9:11] Guilhem Bichot
Yes, having the output of "bzr version-info", or at least "bzr revision-info", in the MANIFEST would be useful. Use case: some QA dev downloads a non-debug snapshot of the server (from pb2 or other places), finds a bug; the server dev needs to know the bzr revision id of this snapshot in order to recreate the corresponding source tree (which she/he can then compile with debugging flags).
The bug reporter can also be a community person who downloaded a nightly snapshot from somewhere.
[2 Nov 2010 10:16] Axel Schwenke
there is related bug #56864 asking to put cmake build information into mysqlbug
[2 Nov 2010 10:57] Jonathan Perkin
Joerg, we recently discussed including CMakeCache.txt which will provide most of this information, could you take a look?  Thanks.
[9 Nov 2010 19:23] Konstantin Osipov
mysqlbug was removed in 5.5.
Thus, Davi's plea should not affect 5.1 or 5.0.

Suggest to have a small script or just a text file containing the necessary information.
[2 Dec 2010 16:14] Sveta Smirnova
Bug #58661 was marked as duplicate of this one.
[21 Dec 2010 10:11] Bugs System
A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/127367

3209 Joerg Bruehe	2010-12-21
      Fix bug#42969    Please add a MANIFEST to each build
      
      This is work-in-progress, not yet ready for review:
      The current changeset is the first step only, but it must
      be committed in order to proceed ("bzr export" is needed).
      
      Step 1: Create "MANIFEST_source" during "make dist".
     @ cmake/make_dist.cmake.in
        Use the new "manifest.cmake" file during "make dist".
     @ cmake/manifest.cmake.in
        New file.
        
        It will contain macros to handle the "MANIFEST*" files
        during various steps of the build,
        the calls will be at other places.
        
        Step 1: Macro "CREATE_MANIFEST_SOURCE" only.
     @ sql/CMakeLists.txt
        Handle the (new) "cmake/manifest.cmake.in" file
        by running CONFIGURE_FILE() on it.
[21 Dec 2010 10:19] Joerg Bruehe
As written in the changeset comment, this first patch is still incomplete - but I cannot proceed without committing it (the new file must become part of "bzr export").

There is no chance to use the same implementation in the autotools-besed 5.1 build and the cmake-based 5.5 build, so the target for this change is 5.5 and up.
[21 Jan 2011 20:41] Bugs System
A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/129366

3209 Joerg Bruehe	2011-01-21
      Fix bug#42969    Please add a MANIFEST to each build
      
      With this change, there will be new files "VERSION_src"
      and "VERSION_bin", which describe the source and the
      binaries.
      They will be contained in all packages.
      
      "VERSION_src" is also part of a source tarball.
      It gives the version as exact as possible, preferably
      by calling "bzr versioninfo" on the source tree.
      If that is not possible, it just contains the three
      level version number.
      
      "VERSION_bin" contains some info when and where the
      binaries were built, the options given to the compiler,
      and the flags controlling the included features.
     @ CMakeLists.txt
        For the new files describing the source and the build
        ("VERSION_src" and "VERSION_bin"), we need a new file
        "cmake/version_files.cmake.in" with the build rules.
        
        1) This file must be configured with the current variables.
        
        2) "VERSION_src" can be created during the cmake phase.
        
        3) "VERSION_bin" must be created during the make phase
           only, because it contains information from files
        which will be written at the end of the cmake phase only.
        Therefore, it must be a custom target which is included
        in all "make" targets.
        
        4) The resulting version files must be included in packages.
     @ cmake/make_dist.cmake.in
        Create a "VERSION_src" file during "make dist".
        
        In case it already exists from a preceding "cmake" run
        (which is quite likely), a new "make dist" must not modify it.
     @ cmake/version_files.cmake.in
        New file.
        
        It will contain macros to create the "VERSION_*" files
        during various steps of the build,
        the calls will be at other places.
        
        1) For source: If running from a BZR tree, always create
           (update) "VERSION_src" by running "bzr version-info".
           Outside a BZR tree, create it only if missing, and
           put the three level version number into it.
        
        2) "VERSION_bin" contains
           - date/time and host name of the build host,
           - information about the platform, if cmake
             provides it
             (2.6.0 does not, the manual is wrong),
           - information about the C and CXX compiler
             and the options given to them
             (these may vary by directory, take those of
             the "sql/" subdirectory),
           - the "WITH*" options (ON or OFF) from
             "CMakeCache.txt"
             (taking them from there rather than using
             "CONFIGURE_FILE()" avoids maintenance when
             features get added or removed).
        
        Several actions are more complicated than is nice,
        this was needed to make it work using cmake 2.6.0.
        When all builds move to newer versions, there will
        hopefully be room for improvement.
     @ support-files/mysql.spec.sh
        Add "VERSION_src" and "VERSION_bin" to the RPM contents.
[24 Jan 2011 11:32] Jonathan Perkin
I don't like the names, they seem counter-intuitive and clash somewhat with the new VERSION file.

How about BZR_INFO and BUILD_INFO?
[31 Jan 2011 19:33] Bugs System
A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/130080

3209 Joerg Bruehe	2011-01-31
      Fix bug#42969    Please add a MANIFEST to each build
      
      With this change, there will be new files "INFO_SRC"
      and "INFO_BIN", which describe the source and the
      binaries.
      They will be contained in all packages:
      - in "tar.gz" and derived packages, in "docs/",
      - in RPMs, in "/usr/share/doc/packages/MySQL-server".
      
      "INFO_SRC" is also part of a source tarball.
      It gives the version as exact as possible, preferably
      by calling "bzr version-info" on the source tree.
      If that is not possible, it just contains the three
      level version number.
      
      "INFO_BIN" contains some info when and where the
      binaries were built, the options given to the compiler,
      and the flags controlling the included features.
      
      (At the time of this writing, January 2011,
      cmake 2.6.4 does not provide some of this desired
      information in the extent promised by its documentation.)
     @ CMakeLists.txt
        For the new files describing the source and the build
        ("INFO_SRC" and "INFO_BIN"), we need a new file
        "cmake/info_macros.cmake.in" with the build rules.
        
        1) This file must be configured with the current variables.
        
        2) "INFO_SRC" can be created during the cmake phase.
        
        3) "INFO_BIN" must be created during the make phase
           only, because it contains information from files
        which will be written at the end of the cmake phase only.
        Therefore, it must be a custom target which is included
        in all "make" targets.
        
        4) The resulting version files must be included in packages.
     @ cmake/info_bin.cmake
        This is the file to create "INFO_BIN",
        by calling the "CREATE_INFO_BIN()" macro.
        
        It must be a separate file, so that the macro
        definitions can be included in other cmake scripts
        without that file inclusion causing a side effect,
        the macro call.
        That call would modify the source tree which should
        be trated read-only.
     @ cmake/info_macros.cmake.in
        This new file contains the macros to create the
        "INFO_*" files during various steps of the build,
        the calls will be at other places.
        
        1) For source: If running from a BZR tree, always create
           (update) "INFO_SRC" by running "bzr version-info".
           Outside a BZR tree, try to take it from exported
           sources, and create it only if missing, in that
           case put the three level version number into it.
        
        2) "INFO_BIN" contains (if provided)
           - date/time and host name of the build host,
           - information about the platform,
             (2.6.4 does not provide it, the manual is wrong),
           - information about the C and CXX compiler
             and the options given to them (Unix only),
           - the feature flags as reported by "cmake -L".
     @ cmake/make_dist.cmake.in
        Create a "VERSION_src" file during "make dist".
        
        In case it already exists from a preceding "cmake" run
        or tree export (which is quite likely), a new
        "make dist" must not modify it.
     @ support-files/mysql.spec.sh
        Add "INFO_SRC" and "INFO_BIN" to the RPM contents.
[1 Feb 2011 13:53] Bugs System
A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/130127

3211 Joerg Bruehe	2011-02-01
      Bug#42969    Please add a MANIFEST to each build
      
      Additional fix:
      cmake variables must be taken using the '@var@' notation
      and then be passed through "configure_file()",
      this was missing for the cmake_[host_]system*" ones.
[7 Feb 2011 12:24] Bugs System
A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/130565

3212 Joerg Bruehe	2011-02-07
      Bug #42969  	Please add a MANIFEST to each build
      
      First attempt at a test that will verify the existence
      of both "INFO_SRC" and "INFO:_BIN", as well as (some of)
      their expected (invariant) contents.
      
      However, this may still need some directory name polishing
      when it is checked on all of our supported platforms.
     @ mysql-test/t/mysql.test
        The patterns searched must match the file creation
        in boh local developer and release builds.
        Path name construction may need changes.
[11 Feb 2011 11:52] Bugs System
A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/131120

3209 Joerg Bruehe	2011-02-11
      Fix bug#42969    Please add a MANIFEST to each build
      
      With this change, there will be new files "INFO_SRC"
      and "INFO_BIN", which describe the source and the
      binaries.
      They will be contained in all packages:
      - in "tar.gz" and derived packages, in "docs/",
      - in RPMs, in "/usr/share/doc/packages/MySQL-server".
      
      "INFO_SRC" is also part of a source tarball.
      It gives the version as exact as possible, preferably
      by calling "bzr version-info" on the source tree.
      If that is not possible, it just contains the three
      level version number.
      
      "INFO_BIN" contains some info when and where the
      binaries were built, the options given to the compiler,
      and the flags controlling the included features.
      
      The tests (test "mysql" in the main suite) are extended
      to verify the existence of both "INFO_SRC" and "INFO_BIN",
      as well as some of the expected contents.
     @ CMakeLists.txt
        For the new files describing the source and the build
        ("INFO_SRC" and "INFO_BIN"), we need a new file
        "cmake/info_macros.cmake.in" with the build rules.
        
        1) This file must be configured with the current variables.
        
        2) "INFO_SRC" can be created during the cmake phase,
           but this should be repeated with each "make" to
        protect against a developer doing only "make" after
        a "bzr pull" (or "bzr commit").
        So have it both as a cmake rule and as a custom target.
        
        3) "INFO_BIN" must be created during the make phase
           only, because it contains information from files
        which will be written at the end of the cmake phase only.
        Therefore, it must be a custom target which is included
        in all "make" targets.
        
        4) The resulting "INFO_*" files must be included in packages.
     @ cmake/info_bin.cmake
        This is the file to create "INFO_BIN",
        by calling the "CREATE_INFO_BIN()" macro.
        
        It must be a separate file, so that the macro
        definitions can be included in other cmake scripts
        without that file inclusion causing a side effect,
        the macro call.
        That call would modify the source tree which should
        be trated read-only.
     @ cmake/info_macros.cmake.in
        This new file contains the macros to create the
        "INFO_*" files during various steps of the build,
        the calls will be at other places.
        
        1) For source: If running from a BZR tree, always create
           (update) "INFO_SRC" by running "bzr version-info".
           Outside a BZR tree, try to take it from exported
           sources, and create it only if missing, in that
           case put the three level version number into it.
        
        2) "INFO_BIN" contains
           - date/time and host name of the build host,
           - information about the platform,
           - information about the C and CXX compiler
             and the options given to them (Unix only),
           - the feature flags as reported by "cmake -L".
     @ cmake/info_src.cmake
        This is the file to create "INFO_SRC",
        by calling the "CREATE_INFO_SRC()" macro.
        
        It must be a separate file, so that the macro
        definitions can be included in other cmake scripts
        without that file inclusion causing a side effect,
        the macro call.
        That call would modify the source tree which should
        be trated read-only.
     @ cmake/make_dist.cmake.in
        Create a "VERSION_src" file during "make dist".
        
        In case it already exists from a preceding "cmake" run
        or tree export (which is quite likely), a new
        "make dist" must not modify it.
     @ mysql-test/r/mysql.result
        Add the expected result of checking the "INFO_*" files.
     @ mysql-test/t/mysql.test
        Check the "INFO_*" files.
        
        The path name to find them is not directly provided
        by MTR, so it is constructed by modifying "libdir".
        
        The names created must match the file placement
        in both local developer and release builds.
        tar.gz and RPM have a different storage layout,
        then even RedHat and SuSE differ, and development
        in BZR is yet another case.
        
        Allow for the different "INFO_BIN" contents
        on Unix vs Windows regarding the compiler.
     @ support-files/mysql.spec.sh
        Add "INFO_SRC" and "INFO_BIN" to the RPM contents.
[5 Apr 2011 12:39] Joerg Bruehe
This has been approved (marked now), and pushed to the main trees.

The fix takes effect for versions 5.5.11, 5.6.2, and their successors:

All packages will contain both an "INFO_SRC" and an "INFO_BIN" file which provide the desired information.
The placement varies by package format, they are put in the same directory as other documentation files.
[14 Apr 2011 13:45] Paul DuBois
Noted in 5.5.11, 5.6.2 changelogs.

MySQL distributions now include an INFO_SRC file that contains
information about the source distribution, such as the MySQL version
from which it was created. MySQL binary distributions additionally
include an INFO_BIN file that contains information about how the
distribution was built, such as compiler options and feature flags.
In RPM packages, these files are located in the
/usr/share/doc/packages/MySQL-server directory. In tar.gz and derived
packages, they are located in the Docs directory under the location
where the distribution is unpacked.

CHANGESET - http://lists.mysql.com/commits/131120