Bug #31442 g_cond_timed_wait_posix_impl: assertion ... filling logfile
Submitted: 8 Oct 2007 12:10 Modified: 12 Jun 12:43
Reporter: Ralf Narozny
Status: Open
Category:Monitoring: Agent Severity:S2 (Serious)
Version:1.2.0.7657,2.0.5.7153 OS:Linux (x86_64-glibc2.3)
Assigned to: Jan Kneschke Target Version:
Tags: agent logfile

[8 Oct 2007 12:10] Ralf Narozny 
Description:
Hi,

on some of our Linux boxes the agent runs amok and writes error messages into the logfile
until the disk is full. This did only happen after a few hours of normal running.

This does not happen on all our 64bit machines. The ones where it happened have InnoDB
disabled, use MEMORY tables and are replication slaves.

Data of the machine:
Linux 2.6.21.1 #1 SMP x86_64 GNU/Linux
MemTotal:      4048120 kB
SwapTotal:     9771576 kB

Logfile content:
...
2007-10-04 20:14:26: (critical) g_cond_timed_wait_posix_impl: assertion
`end_time.tv_nsec < G_NSEC_PER_SEC' failed
2007-10-04 20:14:26: (critical) g_cond_timed_wait_posix_impl: assertion
`end_time.tv_nsec < G_NSEC_PER_SEC' failed
2007-10-04 20:14:26: (critical) g_cond_timed_wait_posix_impl: assertion
`end_time.tv_nsec < G_NSEC_PER_SEC' failed
...

How to repeat:
Install the agent, start it and wait?
[8 Oct 2007 12:14] Ralf Narozny 
Output of SHOW VARIABLES

Attachment: show_variables.txt (text/plain), 15.18 KiB.
[6 Dec 2007 12:55] Jan Kneschke 
the code path looks like:

g_get_current_time(...);
g_time_val_add(...);
g_async_queue_timed_pop(...);

All the calls to g_time_val_add() call g_get_current_time() first.

g_get_current_time() is a wrapper around gettimeofday(). The issue triggered by
g_time_val_add() is a usec (micro-seconds)-value of more than 1 000 000 that should have
been folded into the next second instead.

This can only have 2 issues:
* a ntpd adding microseconds to fix the drift
* a thread-issue, it it looks all clean

I'll add guards before the g_time_val_add() calls and log the values of the assertion and
correct the values before they get into g_time_val_add()
[8 Dec 2007 0:43] Jan Kneschke 
a patch has been committed to trunk in [8507] that implements the guards as discussed
above.
[6 Mar 2008 4:24] Keith Russell 
Patch installed in versions => 1.3.0.8950
[16 Jun 2008 21:17] Paul DuBois 
No changelog entry needed.