Bug #31442 g_cond_timed_wait_posix_impl: assertion ... filling logfile

• Submitted: 8 Oct 2007 10:10
• Modified: 4 Oct 2010 12:41
• Reporter: Ralf Narozny
• Status: Closed
• Category: MySQL Enterprise Monitor: Agent
• Severity: S2 (Serious)
• Version: 1.2.0.7657,2.0.5.7153
• OS: Linux (x86_64-glibc2.3)
• Assigned to: Jan Kneschke
• CPU Architecture: Any
• Tags: agent logfile

[8 Oct 2007 10:10] Ralf Narozny

Description:
Hi,

on some of our Linux boxes the agent runs amok and writes error messages into the logfile until the disk is full. This did only happen after a few hours of normal running.

This does not happen on all our 64bit machines. The ones where it happened have InnoDB disabled, use MEMORY tables and are replication slaves.

Data of the machine:
Linux 2.6.21.1 #1 SMP x86_64 GNU/Linux
MemTotal:      4048120 kB
SwapTotal:     9771576 kB

Logfile content:
...
2007-10-04 20:14:26: (critical) g_cond_timed_wait_posix_impl: assertion
`end_time.tv_nsec < G_NSEC_PER_SEC' failed
2007-10-04 20:14:26: (critical) g_cond_timed_wait_posix_impl: assertion
`end_time.tv_nsec < G_NSEC_PER_SEC' failed
2007-10-04 20:14:26: (critical) g_cond_timed_wait_posix_impl: assertion
`end_time.tv_nsec < G_NSEC_PER_SEC' failed
...

How to repeat:
Install the agent, start it and wait?

[8 Oct 2007 10:14] Ralf Narozny

Output of SHOW VARIABLES

Attachment: show_variables.txt (text/plain), 15.18 KiB.

[6 Dec 2007 11:55] Jan Kneschke

the code path looks like:

g_get_current_time(...);
g_time_val_add(...);
g_async_queue_timed_pop(...);

All the calls to g_time_val_add() call g_get_current_time() first.

g_get_current_time() is a wrapper around gettimeofday(). The issue triggered by g_time_val_add() is a usec (micro-seconds)-value of more than 1 000 000 that should have been folded into the next second instead.

This can only have 2 issues:
* a ntpd adding microseconds to fix the drift
* a thread-issue, it it looks all clean

I'll add guards before the g_time_val_add() calls and log the values of the assertion and correct the values before they get into g_time_val_add()

[7 Dec 2007 23:43] Jan Kneschke

a patch has been committed to trunk in [8507] that implements the guards as discussed above.

[6 Mar 2008 3:24] Keith Russell

Patch installed in versions => 1.3.0.8950

[16 Jun 2008 19:17] Paul DuBois

No changelog entry needed.

[4 Oct 2010 12:41] Mark Leith

This has not been seen for some time now after our patches, closing.