MySQL Bugs: #49707: Files with long names not included in distribution.

Bug #49707	Files with long names not included in distribution.
Submitted:	15 Dec 2009 9:32	Modified:	1 Dec 2016 13:41
Reporter:	Rafal Somla	Email Updates:
Status:	Closed	Impact on me:	None
Category:	MySQL Server: Packaging	Severity:	S3 (Non-critical)
Version:	6.0-backup, 6.0.14, 5.1	OS:	Any
Assigned to:	Kent Boortz	CPU Architecture:	Any

Description:
Some of the files which should be included in the distribution, are skipped because tar complains about too long file names:

> tardir=mysql-6.0.14-alpha && /bin/sh /export/home/pb2/build/sb_0-1075606-1260801913.17/mysql-6.0-backup-export-761430/missing --run tar chof - "$tardir" | GZIP=--best gzip -c >mysql-6.0.14-alpha.tar.gz
> tar: mysql-6.0.14-alpha/mysql-test/suite/backup_extra/flightstats_stress/run_flightstat_stress_with_restore.txt: file name is too long (max 99); not dumped
> tar: mysql-6.0.14-alpha/mysql-test/suite/backup_extra/flightstats_stress/r/flightstat_stress_delete.result: file name is too long (max 99); not dumped
> tar: mysql-6.0.14-alpha/mysql-test/suite/backup_extra/flightstats_stress/r/flightstat_stress_insert.result: file name is too long (max 99); not dumped
> tar: mysql-6.0.14-alpha/mysql-test/suite/backup_extra/flightstats_stress/r/flightstat_stress_update.result: file name is too long (max 99); not dumped
> tar: mysql-6.0.14-alpha/mysql-test/suite/backup_extra/flightstats_stress/r/flightstat_stress_restore.result: file name is too long (max 99); not dumped
> tar: Exiting with failure status due to previous errors
> gtar: mysql-6.0.14-alpha/mysql-test/suite/backup_extra/flightstats_stress/run_flightstat_stress_with_restore.txt: file name is too long (max 99); not dumped
> gtar: mysql-6.0.14-alpha/mysql-test/suite/backup_extra/flightstats_stress/r/flightstat_stress_delete.result: file name is too long (max 99); not dumped
> gtar: mysql-6.0.14-alpha/mysql-test/suite/backup_extra/flightstats_stress/r/flightstat_stress_insert.result: file name is too long (max 99); not dumped
> gtar: mysql-6.0.14-alpha/mysql-test/suite/backup_extra/flightstats_stress/r/flightstat_stress_update.result: file name is too long (max 99); not dumped
> gtar: mysql-6.0.14-alpha/mysql-test/suite/backup_extra/flightstats_stress/r/flightstat_stress_restore.result: file name is too long (max 99); not dumped
> gtar: Exiting with failure status due to previous errors

I observed it in PB2, when the source archive is created. The above files were skipped, but PB2 did not notice that fact. Then it proceed using incomplete distribution which lead to failures in the restore_stress test which is using these files.

One problem is that creating distribution package can fail if some paths are too long. Another problem is that PB2 ignores that fact.

How to repeat:
Do "make dist". Can depend on the version of tar which is used.

Note: I think the problem is not exactly long file names, but long file paths, which is worse. In the above error messages, the limit is 99 characters, so only the full path exceeds that limit.

Thank you for the report.

Verified as described.

Is this verified in our release binaries? If not, I do not understand the tag.

/Staale

The issue should be fixed, but perhaps there is one "flightstats_stress" too many here:

    flightstats_stress/r/flightstat_stress_update.result

;)

There is no real reason to limit our source or binary TAR packages to 99 characters any longer as most systems comes with a TAR implementation that supports the USTAR format. But when doing this change we have to be careful making sure the resulting packages are compatible.

In automake 1.9 and up, there are support to specify to use USTAR format for "make dist", i.e. when creating the source package, you specify in "configure.in" (when the rest of the macros is updated to more modern style, as is in the "mysql-pe" bzr tree)

  AM_INIT_AUTOMAKE([1.9 tar-ustar])

Now, the resulting configure script will favor GNU tar even if not first in PATH, causing some trouble. As it turns out the USTAR format created by GNU tar is not always compatible with other USTAR implementations. In fact, a small test showed these operating systems default TAR not to be able to handle the GNU tar USTAR format

  AIX 5.3 and i5/OS
  Solaris, all versions
  HP-UX 11.11, 11.23, 11.31
  SCO OpenServer 6

This was discovered during this small test. First two TAR files were created, using this script

  # Play nice, a bit under the limits, 98 + 20 + 77
  a=aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
  b=bbbbbbbbbbbbbbbbbbbb
  c=ccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc

  perl -e "print length('$a'), ' '"
  perl -e "print length('$b'), ' '"
  perl -e "print length('$c'), ' '"
  echo ""

  mkdir -p $a/$b
  touch $a/$b/$c

  /opt/csw/bin/gtar --version

  # Don't know how to make Solaris TAR display version,
  # but it is part of the core package, and I can display
  # the version of that one
  echo ""
  echo "Solaris 10 TAR version (SUNWcsu)"
  pkginfo -l SUNWcsu | grep VERSION

  set -x

  /opt/csw/bin/gtar --format=ustar -cf gnu.tar $a
  /usr/bin/tar -cf solaris.tar $a

  find aaaaaaa*

  /opt/csw/bin/gtar -tf gnu.tar
  /usr/bin/tar -tf solaris.tar

  set +x

  rm -fr aaaaaaa*

This script was run on a Solaris 10 machine.

Then the resulting TAR files were copied to various hosts /tmp/ to be unpacked, the TAR archive created using Solaris TAR could unpack correctly on all hosts. The archive created using GNU TAR failed  on the operating systems listed above. GNU tar 1.22 was used.

On the hosts where the unpack doesn't work, only the top directory is created, nothing in it.

Got a hint from the <automake@gnu.org> mailing list that you can set the preferred TAR command to by assigning the path or name to the 'am_cv_prog_tar_ustar' variable right before the AM_INIT_AUTOMAKE macro, or from the command line like

  % ./configure am_cv_prog_tar_ustar=....

Posted by developer:
 
All required files are now included in distribution.