Bug #54780 WARNING: timerHandlingLab now: 25530672 sent: 25529707 diff: 965
Submitted: 24 Jun 2010 14:37 Modified: 6 Jul 2010 8:46
Reporter: lee zhenhua Email Updates:
Status: Not a Bug Impact on me:
None 
Category:MySQL Cluster: Cluster (NDB) storage engine Severity:S2 (Serious)
Version:mysql version 5.1.35 ,ndb-7.0.7 OS:Other (centos 5.2 x86_64)
Assigned to: CPU Architecture:Any
Tags: WARNING: timerHandlingLab now: 25530672 sent: 25529707 diff: 965

[24 Jun 2010 14:37] lee zhenhua
Description:
hello

This is the data node ndb_X_out.log.Why are there so many alarm, these alarm is what reason. What should I do with it.

Thanks.

recvlock waiting for lock, contentions: 3 spins: 3986
completing gcp 346/30 in execTAKE_OVERTCCONF
2010-06-24 22:28:48 [ndbd] WARNING  -- Ndb kernel thread 1 is stuck in: Performing Send elapsed=100
2010-06-24 22:28:48 [ndbd] INFO     -- Watchdog: User time: 51944  System time: 30537
2010-06-24 22:28:48 [ndbd] WARNING  -- Ndb kernel thread 1 is stuck in: Performing Send elapsed=200
2010-06-24 22:28:48 [ndbd] INFO     -- Watchdog: User time: 51944  System time: 30539
2010-06-24 22:28:48 [ndbd] WARNING  -- Ndb kernel thread 1 is stuck in: Performing Send elapsed=300
2010-06-24 22:28:48 [ndbd] INFO     -- Watchdog: User time: 51944  System time: 30558
2010-06-24 22:28:48 [ndbd] WARNING  -- Ndb kernel thread 1 is stuck in: Performing Send elapsed=400
2010-06-24 22:28:48 [ndbd] INFO     -- Watchdog: User time: 51944  System time: 30567
2010-06-24 22:28:48 [ndbd] WARNING  -- Ndb kernel thread 1 is stuck in: Performing Send elapsed=500
2010-06-24 22:28:48 [ndbd] INFO     -- Watchdog: User time: 51944  System time: 30579
2010-06-24 22:28:49 [ndbd] WARNING  -- Ndb kernel thread 1 is stuck in: Performing Send elapsed=600
2010-06-24 22:28:49 [ndbd] INFO     -- Watchdog: User time: 51944  System time: 30599
2010-06-24 22:28:49 [ndbd] WARNING  -- Ndb kernel thread 1 is stuck in: Performing Send elapsed=700
2010-06-24 22:28:49 [ndbd] INFO     -- Watchdog: User time: 51944  System time: 30613
2010-06-24 22:28:49 [ndbd] WARNING  -- Ndb kernel thread 1 is stuck in: Performing Send elapsed=799
2010-06-24 22:28:49 [ndbd] INFO     -- Watchdog: User time: 51944  System time: 30629
2010-06-24 22:28:49 [ndbd] WARNING  -- Ndb kernel thread 1 is stuck in: Performing Send elapsed=900
2010-06-24 22:28:49 [ndbd] INFO     -- Watchdog: User time: 51944  System time: 30649
2010-06-24 22:28:49 [ndbd] WARNING  -- Ndb kernel thread 1 is stuck in: Performing Send elapsed=1000
2010-06-24 22:28:49 [ndbd] INFO     -- Watchdog: User time: 51944  System time: 30669
2010-06-24 22:28:49 [ndbd] WARNING  -- Ndb kernel thread 1 is stuck in: Performing Send elapsed=1100
2010-06-24 22:28:49 [ndbd] INFO     -- Watchdog: User time: 51944  System time: 30689
2010-06-24 22:28:49 [ndbd] WARNING  -- Ndb kernel thread 1 is stuck in: Performing Send elapsed=1200
2010-06-24 22:28:49 [ndbd] INFO     -- Watchdog: User time: 51944  System time: 30706
2010-06-24 22:28:49 [ndbd] WARNING  -- Ndb kernel thread 1 is stuck in: Performing Send elapsed=1300
2010-06-24 22:28:49 [ndbd] INFO     -- Watchdog: User time: 51944  System time: 30719
2010-06-24 22:28:49 [ndbd] WARNING  -- Ndb kernel thread 1 is stuck in: Performing Send elapsed=1400
2010-06-24 22:28:49 [ndbd] INFO     -- Watchdog: User time: 51944  System time: 30739
2010-06-24 22:29:19 [ndbd] WARNING  -- Ndb kernel thread 0 is stuck in: Performing Send elapsed=100
2010-06-24 22:29:19 [ndbd] INFO     -- Watchdog: User time: 52340  System time: 31176
2010-06-24 22:29:19 [ndbd] WARNING  -- Ndb kernel thread 0 is stuck in: Performing Send elapsed=200
2010-06-24 22:29:19 [ndbd] INFO     -- Watchdog: User time: 52340  System time: 31209
2010-06-24 22:29:20 [ndbd] WARNING  -- Ndb kernel thread 0 is stuck in: Performing Send elapsed=299
2010-06-24 22:29:20 [ndbd] INFO     -- Watchdog: User time: 52340  System time: 31239
2010-06-24 22:29:20 [ndbd] WARNING  -- Ndb kernel thread 0 is stuck in: Performing Send elapsed=399
2010-06-24 22:29:20 [ndbd] INFO     -- Watchdog: User time: 52340  System time: 31274
2010-06-24 22:29:20 [ndbd] WARNING  -- Ndb kernel thread 0 is stuck in: Performing Send elapsed=499
2010-06-24 22:29:20 [ndbd] INFO     -- Watchdog: User time: 52340  System time: 31312
2010-06-24 22:29:20 [ndbd] WARNING  -- Ndb kernel thread 0 is stuck in: Performing Send elapsed=599
2010-06-24 22:29:20 [ndbd] INFO     -- Watchdog: User time: 52340  System time: 31342
2010-06-24 22:29:20 [ndbd] WARNING  -- Ndb kernel thread 0 is stuck in: Performing Send elapsed=699
2010-06-24 22:29:20 [ndbd] INFO     -- Watchdog: User time: 52340  System time: 31373
2010-06-24 22:29:20 [ndbd] WARNING  -- Ndb kernel thread 0 is stuck in: Performing Send elapsed=799
2010-06-24 22:29:20 [ndbd] INFO     -- Watchdog: User time: 52340  System time: 31397
2010-06-24 22:29:20 [ndbd] WARNING  -- Ndb kernel thread 0 is stuck in: Performing Send elapsed=899
2010-06-24 22:29:20 [ndbd] INFO     -- Watchdog: User time: 52340  System time: 31434
2010-06-24 22:29:20 [ndbd] WARNING  -- Ndb kernel thread 0 is stuck in: Performing Send elapsed=999
2010-06-24 22:29:20 [ndbd] INFO     -- Watchdog: User time: 52340  System time: 31472
2010-06-24 22:29:20 [ndbd] WARNING  -- Ndb kernel thread 0 is stuck in: Performing Send elapsed=1099
2010-06-24 22:29:20 [ndbd] INFO     -- Watchdog: User time: 52340  System time: 31506
2010-06-24 22:29:20 [ndbd] WARNING  -- Ndb kernel thread 0 is stuck in: Performing Send elapsed=1199
2010-06-24 22:29:20 [ndbd] INFO     -- Watchdog: User time: 52340  System time: 31546
2010-06-24 22:29:21 [ndbd] WARNING  -- Ndb kernel thread 0 is stuck in: Performing Send elapsed=1299
2010-06-24 22:29:21 [ndbd] INFO     -- Watchdog: User time: 52340  System time: 31586
2010-06-24 22:29:21 [ndbd] WARNING  -- Ndb kernel thread 0 is stuck in: Performing Send elapsed=1399
2010-06-24 22:29:21 [ndbd] INFO     -- Watchdog: User time: 52340  System time: 31626
2010-06-24 22:29:21 [ndbd] WARNING  -- Ndb kernel thread 0 is stuck in: Performing Send elapsed=1499
2010-06-24 22:29:21 [ndbd] INFO     -- Watchdog: User time: 52340  System time: 31666
2010-06-24 22:29:21 [ndbd] WARNING  -- Ndb kernel thread 0 is stuck in: Performing Send elapsed=1599
2010-06-24 22:29:21 [ndbd] INFO     -- Watchdog: User time: 52340  System time: 31706
2010-06-24 22:29:21 [ndbd] WARNING  -- Ndb kernel thread 0 is stuck in: Performing Send elapsed=1699
2010-06-24 22:29:21 [ndbd] INFO     -- Watchdog: User time: 52340  System time: 31746
2010-06-24 22:29:21 [ndbd] WARNING  -- Ndb kernel thread 0 is stuck in: Performing Send elapsed=1799
2010-06-24 22:29:21 [ndbd] INFO     -- Watchdog: User time: 52340  System time: 31778
2010-06-24 22:29:21 [ndbd] WARNING  -- Ndb kernel thread 0 is stuck in: Performing Send elapsed=1899
2010-06-24 22:29:21 [ndbd] INFO     -- Watchdog: User time: 52340  System time: 31808
2010-06-24 22:29:21 [ndbd] WARNING  -- Ndb kernel thread 0 is stuck in: Performing Send elapsed=1999
2010-06-24 22:29:21 [ndbd] INFO     -- Watchdog: User time: 52340  System time: 31838

How to repeat:

2010-06-24 22:30:29 [ndbd] WARNING  -- Ndb kernel thread 0 is stuck in: 

Performing Send WARNING: timerHandlingLab now: 25462833 sent: 25460708 diff: 2125
2010-06-24 22:30:13 [ndbd] WARNING  -- Ndb kernel thread 0 is stuck in: Performing Send elapsed=99
2010-06-24 22:30:14 [ndbd] INFO     -- Watchdog: User time: 53484  System time: 33104
2010-06-24 22:30:14 [ndbd] INFO     -- Watchdog: User time: 53484  System time: 33124
2010-06-24 22:30:14 [ndbd] WARNING  -- Watchdog: Warning overslept 342 ms, expected 100 ms.
2010-06-24 22:30:14 [ndbd] WARNING  -- Ndb kernel thread 0 is stuck in: Performing Send elapsed=442
2010-06-24 22:30:14 [ndbd] INFO     -- Watchdog: User time: 53484  System time: 33124
2010-06-24 22:30:14 [ndbd] WARNING  -- Ndb kernel thread 0 is stuck in: Performing Send elapsed=542
2010-06-24 22:30:14 [ndbd] INFO     -- Watchdog: User time: 53484  System time: 33142
2010-06-24 22:30:14 [ndbd] WARNING  -- Ndb kernel thread 0 is stuck in: Performing Send elapsed=642
2010-06-24 22:30:14 [ndbd] INFO     -- Watchdog: User time: 53484  System time: 33143
WARNING: timerHandlingLab now: 25515619 sent: 25514883 diff: 736
2010-06-24 22:30:28 [ndbd] WARNING  -- Ndb kernel thread 0 is stuck in: Performing Send elapsed=100
2010-06-24 22:30:28 [ndbd] INFO     -- Watchdog: User time: 53746  System time: 33446
2010-06-24 22:30:28 [ndbd] WARNING  -- Ndb kernel thread 0 is stuck in: Performing Send elapsed=199
2010-06-24 22:30:28 [ndbd] INFO     -- Watchdog: User time: 53746  System time: 33460
2010-06-24 22:30:28 [ndbd] WARNING  -- Ndb kernel thread 0 is stuck in: Performing Send elapsed=299
2010-06-24 22:30:28 [ndbd] INFO     -- Watchdog: User time: 53746  System time: 33500
2010-06-24 22:30:29 [ndbd] WARNING  -- Ndb kernel thread 0 is stuck in: Performing Send elapsed=400
2010-06-24 22:30:29 [ndbd] INFO     -- Watchdog: User time: 53746  System time: 33527
2010-06-24 22:30:29 [ndbd] WARNING  -- Ndb kernel thread 0 is stuck in: Performing Send elapsed=500
2010-06-24 22:30:29 [ndbd] INFO     -- Watchdog: User time: 53746  System time: 33537
elapsed=600
2010-06-24 22:30:29 [ndbd] INFO     -- Watchdog: User time: 53746  System time: 33547
2010-06-24 22:30:29 [ndbd] WARNING  -- Ndb kernel thread 0 is stuck in: Performing Send elapsed=700
2010-06-24 22:30:29 [ndbd] INFO     -- Watchdog: User time: 53746  System time: 33557
2010-06-24 22:30:29 [ndbd] WARNING  -- Ndb kernel thread 0 is stuck in: Performing Send elapsed=800
2010-06-24 22:30:29 [ndbd] INFO     -- Watchdog: User time: 53746  System time: 33566
2010-06-24 22:30:29 [ndbd] WARNING  -- Ndb kernel thread 0 is stuck in: Performing Send elapsed=900
2010-06-24 22:30:29 [ndbd] INFO     -- Watchdog: User time: 53746  System time: 33576
WARNING: timerHandlingLab now: 25530672 sent: 25529707 diff: 965
2010-06-24 22:30:30 [ndbd] WARNING  -- Ndb kernel thread 0 is stuck in: Performing Send elapsed=100
2010-06-24 22:30:30 [ndbd] INFO     -- Watchdog: User time: 53764  System time: 33598
2010-06-24 22:30:30 [ndbd] WARNING  -- Ndb kernel thread 0 is stuck in: Performing Send elapsed=200
2010-06-24 22:30:30 [ndbd] INFO     -- Watchdog: User time: 53764  System time: 33610
WARNING: timerHandlingLab now: 25531473 sent: 25531208 diff: 265WARNING: timerHandlingLab now: 25531473 sent: 25531208 diff: 265

This is how I what alarm, thanks.
[6 Jul 2010 8:46] Andrew Hutchings
Hello Lee,

This is a load or latency problem inside the server.  If you are using Xeon processors with Nehalem cores please disable NUMA in the kernel (boot with numa=off).  Otherwise you will need to look at your configuration.