Bug #47305 Test backup.backup_log_filesize fails on gcov builds
Submitted: 14 Sep 2009 10:24 Modified: 27 Oct 2009 13:02
Reporter: Ingo Strüwing Email Updates:
Status: Closed Impact on me:
None 
Category:Tools: MTR / mysql-test-run Severity:S7 (Test Cases)
Version:6.0 OS:Linux (gcov build on x86_64)
Assigned to: Bjørn Munch CPU Architecture:Any
Tags: pbfail pushbuild environment teamtree

[14 Sep 2009 10:24] Ingo Strüwing
Description:
I did already try to resolve this by sending email to the release engineers. But I have not seen a reply so far.

I can repeat the problem on my machine, Ubuntu 64-bit. The stack trace shows that mysqltest crashes in getenv(), called from gcov_exit(), called from exit(). So this is most probaby a libc problem.

However, the crash can be avoided by exporting the environment variable GCOV_PREFIX with an empty value.

Now my question is: Is there a way to set and export the environment variable for a test run? Are the statements in mysql-test/collections/*.{push|daily|weekly} interpreted by a shell, so that adding "GCOV_PREFIX= " would be converted into an environment assignment and temoprary export? Is there a more general way to set that variable? E.g. in MTR? Or can it be done by pushbuild on 64-bit platforms that do gcov builds?

Note that the problem exists on the backup team tree only at the moment. It will most probably prevent a push of that tree to a staging tree.

How to repeat:
http://pb2.norway.sun.com/web.py?template=mysql_show_test_failure&test_failure_id=2315383&...
[14 Sep 2009 10:30] Ingo Strüwing
Medium defect - no known way to add an environment variable to a bushbuild run.
No known workaround for pushbuild runs.
Substantial impact - prevents push of backup.
[29 Sep 2009 13:17] Daniel Fischer
To answer the question, yes I can set an environment variable for gcov runs. But this isn't the problem.

Running the test with --valgrind-mysqltest on a binary with gcov support reveals:
==28277== Invalid read of size 2
==28277==    at 0x41F9591: getenv (in /lib/tls/i686/cmov/libc-2.8.90.so)
==28277==    by 0x82C56F7: gcov_exit (in /home/df/tmp/mysql-5.4.4-alpha/client/mysqltest)
==28277==    by 0x41F9D88: exit (in /lib/tls/i686/cmov/libc-2.8.90.so)
==28277==    by 0x8091F3C: _ZL16cleanup_and_exiti (mysqltest.cc:1189)
==28277==    by 0x80C1D73: main (mysqltest.cc:8180)
==28277==  Address 0x4b8fc58 is 24 bytes inside a block of size 105 free'd
==28277==    at 0x4024B4A: free (vg_replace_malloc.c:323)
==28277==    by 0x80E9CB9: _myfree (safemalloc.c:326)
==28277==    by 0x80895E3: var_free(void*) (mysqltest.cc:1953)
==28277==    by 0x81074AA: my_hash_free_elements (hash.c:114)
==28277==    by 0x8107334: my_hash_free (hash.c:135)
==28277==    by 0x809185A: free_used_memory() (mysqltest.cc:1132)
==28277==    by 0x8091CE6: _ZL16cleanup_and_exiti (mysqltest.cc:1169)
==28277==    by 0x80C1D73: main (mysqltest.cc:8180)

Which indicates that we are freeing memory that is part of the environment somewhere.

Closer inspection reveals that mysqltest, when assigning a value to an environment variable with LET, stores the entire KEY=VALUE string in the variable structure (in env_s). In order to write the variable to the environment, putenv() is called with the same string as argument. Later, when variables are freed, the memory pointed to by env_s is also freed. 

The behaviour of putenv differs between platforms. On platforms such as Mac OS X, or Linux with glibc 2.1.2 or higher, on the latter of which we run gcov on, the string passed to putenv becomes part of the environment. On these platforms, freeing the memory pointed to by env_s is a bug, albeit a usually harmless one because exit is called directly after the memory is freed. When running under gcov, however, the environment is accessed after exit is called and accessing memory that is no longer available during a search for GCOV_PREFIX is what triggers the bug. Setting GCOV_PREFIX shortcuts the search, and prevents the bug from being triggered.

On some other platforms (notably glibc before 2.1.2 and 4.4BSD), a copy of the string becomes part of the environment. On these platforms, not freeing env_s would cause a memory leak, albeit a very short-lived one as exit is called immediately afterwards anyway.

My conclusion is that it would be best not to free v->env_s in mysqltest.cc:var_free().

--- mysqltest.cc	2009-09-28 11:29:10.000000000 +0200
+++ mysqltest.cc	2009-09-29 15:16:42.000000000 +0200
@@ -1946,7 +1946,6 @@
 void var_free(void *v)
 {
   my_free(((VAR*) v)->str_val, MYF(MY_WME));
-  my_free(((VAR*) v)->env_s, MYF(MY_WME|MY_ALLOW_ZERO_PTR));
   if (((VAR*)v)->alloced)
     my_free(v, MYF(MY_WME));
 }
[29 Sep 2009 14:30] Ingo Strüwing
In that case it might even be better to allocate all strings that go to the environment with my_once_alloc(). That way they won't count as leaks in debug builds (which use safemalloc()).

We probably can't trick valgrind in a similar way? But at least we could define an exclusion rule.
[1 Oct 2009 15:04] Bugs System
A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/85403

2876 Bjorn Munch	2009-10-01
      Bug #47305 Test backup.backup_log_filesize fails on gcov builds
      Crashes in getenv()
      Don't free string sent to putenv, allocate with "once"
[2 Oct 2009 18:49] Ingo Strüwing
The patch works. Approved by me. IMHO another review is not required. I suggest to let Lars decide.
[5 Oct 2009 8:23] Bugs System
A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/85708

2861 Bjorn Munch	2009-10-05
      Bug #47305 Test backup.backup_log_filesize fails on gcov builds
      Crashes in getenv()
      Don't free string sent to putenv, allocate with "once"
[5 Oct 2009 10:13] Bjørn Munch
Both reviewers ticked the box
[5 Oct 2009 12:01] Bjørn Munch
Pushed to mysql-6.0-codebase-mtr
[17 Oct 2009 9:11] Rafal Somla
Unfortunately, this fix introduced bug#47851. The problem is that the code does free memory now allocated with my_once_strdup().
[17 Oct 2009 9:35] Bugs System
A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/87229

2878 Rafal Somla	2009-10-17
      Remove fix for BUG#47305 because it causes BUG#47851.
[22 Oct 2009 20:17] Bugs System
Pushed into 6.0.14-alpha (revid:alik@sun.com-20091022201524-0efl2fzebfuuf0vk) (version source revid:bjorn.munch@sun.com-20091005082215-f1olshi6ao2954ch) (merge vers: 6.0.14-alpha) (pib:13)
[22 Oct 2009 23:02] Paul DuBois
Test suite change. No changelog entry needed.

Setting report to NDI pending push into 5.5.x.
[25 Oct 2009 13:38] Bugs System
Pushed into 6.0.14-alpha (revid:alik@sun.com-20091025133616-ca4inerav4vpdnaz) (version source revid:rafal.somla@sun.com-20091017093435-re0ajk0zuubncqg6) (merge vers: 6.0.14-alpha) (pib:13)
[25 Oct 2009 20:55] Paul DuBois
Setting report to NDI pending push into 5.5.x.
[27 Oct 2009 13:02] Bjørn Munch
Version was set wrong; the fixed code is in 6.0-codebase but not in "lower" branches, so the fix is irrelevant there.
[26 Dec 2009 8:47] Sergei Golubchik
Reverted. Memory allocated with my_once_malloc is free'd on exit, that is my_once_malloc is wrong choice for memory that should be never free'd.

Duplicate of Bug#48758