MySQL Bugs: #31733: mysqldump linux: _dl_open: Assertion `_dl_debug

Bug #31733	mysqldump linux: _dl_open: Assertion `_dl_debug_initialize ...` failed
Submitted:	21 Oct 2007 3:40	Modified:	17 Jun 2009 13:26
Reporter:	Timothy Smith	Email Updates:
Status:	Unsupported	Impact on me:	None
Category:	Tools: Pushbuild	Severity:	S2 (Serious)
Version:	5.1, 5.2	OS:	Linux
Assigned to:	Daniel Fischer	CPU Architecture:	Any
Tags:	assertion, pbfail, pushbuild, test failure
Triage:	Triaged: D1 (Critical)

Description:
This is a glibc race condition bug.  I'm filing it as a pointer for anyone searching the bugs database.  The solution is to upgrade glibc.

The important error message is:

Inconsistency detected by ld.so: dl-open.c: 610: _dl_open: Assertion `_dl_debug_initialize (0, args.nsid)->r_state == RT_CONSISTENT' failed!

They symptom in the test suite (for the mysqldump.test failures) is:

mysqltest: At line 1619: command "$MYSQL_IMPORT --silent --use-threads=2 test $MYSQLTEST_VARDIR/tmp/t1.txt $MYSQLTEST_VARDIR/tmp/t2.txt $MYSQLTEST_VARDIR/std_data_ln/words.dat $MYSQLTEST_VARDIR/std_data_ln/words2.dat" failed

How to repeat:
This is a very rare failure in the test suite.  It appears just 24 times in the PushBuild database.  21 of those happen with mysqldump.test; the other three are ndb_dd_backuprestore, ndb_single_user, and rpl_stm_insertdelayed.  It appears on platforms sapsrv1, sapsrv2, and sles10-ia64-a-2.

Probably will be next to impossible to repeat just by running the test.

https://intranet.mysql.com/secure/pushbuild/xref.pl?startdate=&enddate=&dir=&plat=&testtyp...

Suggested fix:
Based on what Google says, upgrade glibc.  See the discussion here, for example:

http://sourceware.org/ml/libc-alpha/2006-10/msg00080.html

This has now failed 64 times and counting in pushbuild. Any idea when the pb machines will upgrade their glibc?

It's really bad to have sporadic failures like this one. It consumes developer time if we have to investigate strange errors that happen in our push and not in the previous push.

There seems to be two glibc bugs for this:

    Assertion `...r_state == RT_CONSISTENT' failed!
    <http://sources.redhat.com/bugzilla/show_bug.cgi?id=4578>

    Race in _dl_open with r_debug.r_state consistency check
    <http://sources.redhat.com/bugzilla/show_bug.cgi?id=3429>

At this time, the first one is not released, while the second one, which looks like the more likely candidate, was released with glibc 2.5.

Bug#43077 has been marked as a duplicate of this one.

Setting to unsupported because it only affects one PB1 host (which we can't upgrade to a version of glibc with the fix).