MySQL Bugs: #31733: mysqldump linux: _dl_open: Assertion `_dl_debug

Bug #31733	mysqldump linux: _dl_open: Assertion `_dl_debug_initialize ...` failed
Submitted:	21 Oct 2007 5:40	Modified:	17 Jun 15:26
Reporter:	Timothy Smith
Status:	Unsupported
Category:	Tools: Pushbuild	Severity:	S2 (Serious)
Version:	5.1, 5.2	OS:	Linux
Assigned to:	Daniel Fischer	Target Version:	5.1+
Tags:	assertion, test failure, pushbuild, pbfail
Triage:	Triaged: D1 (Critical)

Description:
This is a glibc race condition bug.  I'm filing it as a pointer for anyone searching the
bugs database.  The solution is to upgrade glibc.

The important error message is:

Inconsistency detected by ld.so: dl-open.c: 610: _dl_open: Assertion `_dl_debug_initialize
(0, args.nsid)->r_state == RT_CONSISTENT' failed!

They symptom in the test suite (for the mysqldump.test failures) is:

mysqltest: At line 1619: command "$MYSQL_IMPORT --silent --use-threads=2 test
$MYSQLTEST_VARDIR/tmp/t1.txt $MYSQLTEST_VARDIR/tmp/t2.txt
$MYSQLTEST_VARDIR/std_data_ln/words.dat $MYSQLTEST_VARDIR/std_data_ln/words2.dat" failed

How to repeat:
This is a very rare failure in the test suite.  It appears just 24 times in the PushBuild
database.  21 of those happen with mysqldump.test; the other three are
ndb_dd_backuprestore, ndb_single_user, and rpl_stm_insertdelayed.  It appears on platforms
sapsrv1, sapsrv2, and sles10-ia64-a-2.

Probably will be next to impossible to repeat just by running the test.

https://intranet.mysql.com/secure/pushbuild/xref.pl?startdate=&enddate=&dir=&plat=&testtyp
e=&testname=&testtext=Inconsistency+detected+by+ld.so&limit=50

Suggested fix:
Based on what Google says, upgrade glibc.  See the discussion here, for example:

http://sourceware.org/ml/libc-alpha/2006-10/msg00080.html

This has now failed 64 times and counting in pushbuild. Any idea when the pb machines will
upgrade their glibc?

It's really bad to have sporadic failures like this one. It consumes developer time if we
have to investigate strange errors that happen in our push and not in the previous push.

There seems to be two glibc bugs for this:

    Assertion `...r_state == RT_CONSISTENT' failed!
    <http://sources.redhat.com/bugzilla/show_bug.cgi?id=4578>

    Race in _dl_open with r_debug.r_state consistency check
    <http://sources.redhat.com/bugzilla/show_bug.cgi?id=3429>

At this time, the first one is not released, while the second one, which looks like the
more likely candidate, was released with glibc 2.5.

Bug#43077 has been marked as a duplicate of this one.

Setting to unsupported because it only affects one PB1 host (which we can't upgrade to a
version of glibc with the fix).