ushastry@ushastry:~/mybuild/mysql-cluster-com-7.3.2$ gdb --args bin/memcached -E lib/ndb_engine.so GNU gdb (Ubuntu/Linaro 7.4-2012.04-0ubuntu2.1) 7.4-2012.04 Copyright (C) 2012 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-linux-gnu". For bug reporting instructions, please see: ... Reading symbols from /home/ushastry/mybuild/mysql-cluster-com-7.3.2/bin/memcached...done. (gdb) run Starting program: /home/ushastry/mybuild/mysql-cluster-com-7.3.2/bin/memcached -E lib/ndb_engine.so [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1". [New Thread 0x7ffff6176700 (LWP 18849)] 18-Sep-2013 14:22:53 IST NDB Memcache 5.6.11-ndb-7.3.2 started [NDB 7.3.2; MySQL 5.6.11] Contacting primary management server (localhost:1186) ... [Thread 0x7ffff6176700 (LWP 18849) exited] [New Thread 0x7ffff4b4c700 (LWP 18851)] [New Thread 0x7ffff4b2b700 (LWP 18852)] [New Thread 0x7ffff7fe5700 (LWP 18853)] [New Thread 0x7ffff7fc4700 (LWP 18854)] [New Thread 0x7ffff7fa3700 (LWP 18855)] Connected to "localhost:1186" as node id 5. Retrieved 4 key prefixes for server role "default_role". The default behavior is that: GET uses NDB only SET uses NDB only DELETE uses NDB only. The 3 explicitly defined key prefixes are "b:" (demo_table_large), "mc:" () and "t:" (demo_table_tabs) [New Thread 0x7ffff7f55700 (LWP 18857)] [New Thread 0x7ffff7f34700 (LWP 18858)] [New Thread 0x7ffff7f13700 (LWP 18859)] [New Thread 0x7ffff7ef2700 (LWP 18860)] [New Thread 0x7ffff7ed1700 (LWP 18861)] Connected to "127.0.0.1" as node id 6. Server started with 4 threads. Priming the pump ... [New Thread 0x7fffdf1f4700 (LWP 18863)] [New Thread 0x7fffdf1d3700 (LWP 18864)] [New Thread 0x7fffdf1b2700 (LWP 18865)] [New Thread 0x7fffdf191700 (LWP 18866)] [New Thread 0x7fffdf170700 (LWP 18867)] Connected to "127.0.0.1" as node id 7. Scheduler: using 2 connections to cluster 0 [New Thread 0x7ffff6176700 (LWP 18868)] [New Thread 0x7fffdf14f700 (LWP 18869)] [New Thread 0x7fffde94e700 (LWP 18870)] [New Thread 0x7fffde14d700 (LWP 18871)] Scheduler: starting for 1 cluster; c0,f0,g1,t1 done [0.808 sec]. [New Thread 0x7fffdd73f700 (LWP 18872)] [New Thread 0x7fffdcf3e700 (LWP 18873)] [New Thread 0x7fffcffff700 (LWP 18874)] [New Thread 0x7fffcf7fe700 (LWP 18875)] [New Thread 0x7fffceffd700 (LWP 18876)] [New Thread 0x7fffce7fc700 (LWP 18877)] Failed to notify thread: Socket operation on non-socket Failed to notify thread: Socket operation on non-socket Failed to notify thread: Socket operation on non-socket Failed to notify thread: Socket operation on non-socket Failed to notify thread: Socket operation on non-socket Failed to notify thread: Socket operation on non-socket Failed to notify thread: Socket operation on non-socket Failed to notify thread: Socket operation on non-socket memcached: /home/ushastry/mybuild/mysql-cluster-com-7.3.2/storage/ndb/memcache/extra/memcached/daemon/memcached.c:4968: unregister_event: Assertion `c->registered_in_libevent' failed. Program received signal SIGABRT, Aborted. 0x00007ffff70ed425 in raise () from /lib/x86_64-linux-gnu/libc.so.6 (gdb) thread apply all bt Thread 27 (Thread 0x7fffce7fc700 (LWP 18877)): #0 0x00007ffff71ab353 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6 #1 0x00007ffff7999cc7 in epoll_dispatch (base=0x1183790, tv=) at epoll.c:407 #2 0x00007ffff7985af4 in event_base_loop (base=0x1183790, flags=0) at event.c:1607 #3 0x00000000004198f5 in worker_libevent (arg=0x1180df0) at /home/ushastry/mybuild/mysql-cluster-com-7.3.2/storage/ndb/memcache/extra/memcached/daemon/thread.c:304 #4 0x00007ffff7bc4e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #5 0x00007ffff71aacbd in clone () from /lib/x86_64-linux-gnu/libc.so.6 #6 0x0000000000000000 in ?? () Thread 26 (Thread 0x7fffceffd700 (LWP 18876)): #0 0x00007ffff71ab353 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6 #1 0x00007ffff7999cc7 in epoll_dispatch (base=0x1182e00, tv=) at epoll.c:407 #2 0x00007ffff7985af4 in event_base_loop (base=0x1182e00, flags=0) at event.c:1607 #3 0x00000000004198f5 in worker_libevent (arg=0x1180cf8) at /home/ushastry/mybuild/mysql-cluster-com-7.3.2/storage/ndb/memcache/extra/memcached/daemon/thread.c:304 #4 0x00007ffff7bc4e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #5 0x00007ffff71aacbd in clone () from /lib/x86_64-linux-gnu/libc.so.6 #6 0x0000000000000000 in ?? () Thread 25 (Thread 0x7fffcf7fe700 (LWP 18875)): #0 0x00007ffff71ab353 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6 #1 0x00007ffff7999cc7 in epoll_dispatch (base=0x1182470, tv=) at epoll.c:407 #2 0x00007ffff7985af4 in event_base_loop (base=0x1182470, flags=0) at event.c:1607 #3 0x00000000004198f5 in worker_libevent (arg=0x1180c00) at /home/ushastry/mybuild/mysql-cluster-com-7.3.2/storage/ndb/memcache/extra/memcached/daemon/thread.c:304 #4 0x00007ffff7bc4e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #5 0x00007ffff71aacbd in clone () from /lib/x86_64-linux-gnu/libc.so.6 #6 0x0000000000000000 in ?? () Thread 24 (Thread 0x7fffcffff700 (LWP 18874)): #0 0x00007ffff71ab353 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6 #1 0x00007ffff7999cc7 in epoll_dispatch (base=0x1181ae0, tv=) at epoll.c:407 #2 0x00007ffff7985af4 in event_base_loop (base=0x1181ae0, flags=0) at event.c:1607 #3 0x00000000004198f5 in worker_libevent (arg=0x1180b08) at /home/ushastry/mybuild/mysql-cluster-com-7.3.2/storage/ndb/memcache/extra/memcached/daemon/thread.c:304 #4 0x00007ffff7bc4e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #5 0x00007ffff71aacbd in clone () from /lib/x86_64-linux-gnu/libc.so.6 #6 0x0000000000000000 in ?? () Thread 23 (Thread 0x7fffdcf3e700 (LWP 18873)): ---Type to continue, or q to quit--- #0 0x00007ffff71ab353 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6 #1 0x00007ffff7999cc7 in epoll_dispatch (base=0x1181150, tv=) at epoll.c:407 #2 0x00007ffff7985af4 in event_base_loop (base=0x1181150, flags=0) at event.c:1607 #3 0x00000000004198f5 in worker_libevent (arg=0x1180a10) at /home/ushastry/mybuild/mysql-cluster-com-7.3.2/storage/ndb/memcache/extra/memcached/daemon/thread.c:304 #4 0x00007ffff7bc4e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #5 0x00007ffff71aacbd in clone () from /lib/x86_64-linux-gnu/libc.so.6 #6 0x0000000000000000 in ?? () Thread 22 (Thread 0x7fffdd73f700 (LWP 18872)): #0 0x00007ffff7bc90fe in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 #1 0x00007ffff685d839 in NdbCondition_WaitTimeoutAbs (p_cond=0x7fffe8007ce0, p_mutex=0x7fffe8001650, abstime=0x7fffdd73ec50) at /home/ushastry/mybuild/mysql-cluster-com-7.3.2/storage/ndb/src/common/portlib/NdbCondition.c:210 #2 0x00007ffff685d715 in NdbCondition_WaitTimeout (p_cond=0x7fffe8007ce0, p_mutex=0x7fffe8001650, msecs=1000) at /home/ushastry/mybuild/mysql-cluster-com-7.3.2/storage/ndb/src/common/portlib/NdbCondition.c:153 #3 0x00007ffff67bc2e1 in NdbEventBuffer::pollEvents (this=0x7fffe8005520, aMillisecondNumber=1000, latestGCI=0x0) at /home/ushastry/mybuild/mysql-cluster-com-7.3.2/storage/ndb/src/ndbapi/NdbEventOperationImpl.cpp:1258 #4 0x00007ffff67b017a in Ndb::pollEvents (this=0x7fffdd73ed00, aMillisecondNumber=1000, latestGCI=0x0) at /home/ushastry/mybuild/mysql-cluster-com-7.3.2/storage/ndb/src/ndbapi/Ndb.cpp:2014 #5 0x00007ffff679ce93 in server_roles_reload_waiter (conn=0x6478d0, server_role=0x7ffff6911f33 "default_role") at /home/ushastry/mybuild/mysql-cluster-com-7.3.2/storage/ndb/memcache/src/Config_v1.cc:675 #6 0x00007ffff679a390 in Configuration::waitForReconfSignal (this=0x6a6a70) at /home/ushastry/mybuild/mysql-cluster-com-7.3.2/storage/ndb/memcache/src/Configuration.cc:389 #7 0x00007ffff678b266 in run_reconfig_listener_thread (p=0x638570) at /home/ushastry/mybuild/mysql-cluster-com-7.3.2/storage/ndb/memcache/src/ndb_configuration.cc:212 #8 0x00007ffff7bc4e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #9 0x00007ffff71aacbd in clone () from /lib/x86_64-linux-gnu/libc.so.6 #10 0x0000000000000000 in ?? () Thread 21 (Thread 0x7fffde14d700 (LWP 18871)): #0 0x00007ffff7bc90fe in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 #1 0x00007ffff678feb2 in S::Connection::run_ndb_send_thread (this=0x744a40) at /home/ushastry/mybuild/mysql-cluster-com-7.3.2/storage/ndb/memcache/src/schedulers/S_sched.cc:850 #2 0x00007ffff6790638 in run_send_thread (v=0x744a40) at /home/ushastry/mybuild/mysql-cluster-com-7.3.2/storage/ndb/memcache/src/schedulers/S_sched.cc:998 #3 0x00007ffff7bc4e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #4 0x00007ffff71aacbd in clone () from /lib/x86_64-linux-gnu/libc.so.6 #5 0x0000000000000000 in ?? () Thread 20 (Thread 0x7fffde94e700 (LWP 18870)): #0 0x00007ffff71ab353 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6 #1 0x00007ffff683f362 in TransporterRegistry::pollReceive (this=0x70a640, timeOutMillis=10, recvdata=...) ---Type to continue, or q to quit--- at /home/ushastry/mybuild/mysql-cluster-com-7.3.2/storage/ndb/src/common/transporter/TransporterRegistry.cpp:1138 #2 0x00007ffff682eefa in TransporterRegistry::pollReceive (this=0x70a640, timeOutMillis=10) at /home/ushastry/mybuild/mysql-cluster-com-7.3.2/storage/ndb/include/transporter/TransporterRegistry.hpp:609 #3 0x00007ffff6829c4d in TransporterFacade::external_poll (this=0x726b90, wait_time=10) at /home/ushastry/mybuild/mysql-cluster-com-7.3.2/storage/ndb/src/ndbapi/TransporterFacade.cpp:826 #4 0x00007ffff682d58c in TransporterFacade::do_poll (this=0x726b90, clnt=0x744f10, wait_time=10, is_poll_owner=false, stay_poll_owner=false) at /home/ushastry/mybuild/mysql-cluster-com-7.3.2/storage/ndb/src/ndbapi/TransporterFacade.cpp:2277 #5 0x00007ffff681a4a9 in trp_client::do_poll (this=0x744f10, to=10) at /home/ushastry/mybuild/mysql-cluster-com-7.3.2/storage/ndb/src/ndbapi/trp_client.cpp:125 #6 0x00007ffff681aff1 in PollGuard::wait_for_input (this=0x7fffde94dd60, wait_time=10) at /home/ushastry/mybuild/mysql-cluster-com-7.3.2/storage/ndb/src/ndbapi/trp_client.cpp:386 #7 0x00007ffff6822285 in MultiNdbWakeupHandler::waitForInput (this=0x74bea0, _objs=0x744e28, _cnt=0, min_req=1, timeout_millis=1204, nready=0x7fffde94dde8) at /home/ushastry/mybuild/mysql-cluster-com-7.3.2/storage/ndb/src/ndbapi/WakeupHandler.cpp:192 #8 0x00007ffff681e3ab in NdbWaitGroup::wait (this=0x705800, arrayHead=@0x7fffde94de20: 0x0, timeout_millis=5000, min_ndbs=1) at /home/ushastry/mybuild/mysql-cluster-com-7.3.2/storage/ndb/src/ndbapi/NdbWaitGroup.cpp:103 #9 0x00007ffff67903fa in S::Connection::run_ndb_poll_thread (this=0x744a40) at /home/ushastry/mybuild/mysql-cluster-com-7.3.2/storage/ndb/memcache/src/schedulers/S_sched.cc:961 #10 0x00007ffff679065a in run_poll_thread (v=0x744a40) at /home/ushastry/mybuild/mysql-cluster-com-7.3.2/storage/ndb/memcache/src/schedulers/S_sched.cc:1003 #11 0x00007ffff7bc4e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #12 0x00007ffff71aacbd in clone () from /lib/x86_64-linux-gnu/libc.so.6 #13 0x0000000000000000 in ?? () Thread 19 (Thread 0x7fffdf14f700 (LWP 18869)): #0 0x00007ffff7bc90fe in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 #1 0x00007ffff678feb2 in S::Connection::run_ndb_send_thread (this=0x70ef80) at /home/ushastry/mybuild/mysql-cluster-com-7.3.2/storage/ndb/memcache/src/schedulers/S_sched.cc:850 #2 0x00007ffff6790638 in run_send_thread (v=0x70ef80) at /home/ushastry/mybuild/mysql-cluster-com-7.3.2/storage/ndb/memcache/src/schedulers/S_sched.cc:998 #3 0x00007ffff7bc4e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #4 0x00007ffff71aacbd in clone () from /lib/x86_64-linux-gnu/libc.so.6 #5 0x0000000000000000 in ?? () Thread 18 (Thread 0x7ffff6176700 (LWP 18868)): #0 0x00007ffff71ab353 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6 #1 0x00007ffff683f362 in TransporterRegistry::pollReceive (this=0x676830, timeOutMillis=10, recvdata=...) at /home/ushastry/mybuild/mysql-cluster-com-7.3.2/storage/ndb/src/common/transporter/TransporterRegistry.cpp:1138 #2 0x00007ffff682eefa in TransporterRegistry::pollReceive (this=0x676830, timeOutMillis=10) at /home/ushastry/mybuild/mysql-cluster-com-7.3.2/storage/ndb/include/transporter/TransporterRegistry.hpp:609 #3 0x00007ffff6829c4d in TransporterFacade::external_poll (this=0x6cd600, wait_time=10) at /home/ushastry/mybuild/mysql-cluster-com-7.3.2/storage/ndb/src/ndbapi/TransporterFacade.cpp:826 #4 0x00007ffff682d58c in TransporterFacade::do_poll (this=0x6cd600, clnt=0x7049e0, wait_time=10, is_poll_owner=false, stay_poll_owner=false) ---Type to continue, or q to quit--- at /home/ushastry/mybuild/mysql-cluster-com-7.3.2/storage/ndb/src/ndbapi/TransporterFacade.cpp:2277 #5 0x00007ffff681a4a9 in trp_client::do_poll (this=0x7049e0, to=10) at /home/ushastry/mybuild/mysql-cluster-com-7.3.2/storage/ndb/src/ndbapi/trp_client.cpp:125 #6 0x00007ffff681aff1 in PollGuard::wait_for_input (this=0x7ffff6175d60, wait_time=10) at /home/ushastry/mybuild/mysql-cluster-com-7.3.2/storage/ndb/src/ndbapi/trp_client.cpp:386 #7 0x00007ffff6822285 in MultiNdbWakeupHandler::waitForInput (this=0x7057c0, _objs=0x7139e8, _cnt=0, min_req=1, timeout_millis=1204, nready=0x7ffff6175de8) at /home/ushastry/mybuild/mysql-cluster-com-7.3.2/storage/ndb/src/ndbapi/WakeupHandler.cpp:192 #8 0x00007ffff681e3ab in NdbWaitGroup::wait (this=0x6e2b40, arrayHead=@0x7ffff6175e20: 0x0, timeout_millis=5000, min_ndbs=1) at /home/ushastry/mybuild/mysql-cluster-com-7.3.2/storage/ndb/src/ndbapi/NdbWaitGroup.cpp:103 #9 0x00007ffff67903fa in S::Connection::run_ndb_poll_thread (this=0x70ef80) at /home/ushastry/mybuild/mysql-cluster-com-7.3.2/storage/ndb/memcache/src/schedulers/S_sched.cc:961 #10 0x00007ffff679065a in run_poll_thread (v=0x70ef80) at /home/ushastry/mybuild/mysql-cluster-com-7.3.2/storage/ndb/memcache/src/schedulers/S_sched.cc:1003 #11 0x00007ffff7bc4e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #12 0x00007ffff71aacbd in clone () from /lib/x86_64-linux-gnu/libc.so.6 #13 0x0000000000000000 in ?? () Thread 17 (Thread 0x7fffdf170700 (LWP 18867)): #0 0x00007ffff719f303 in poll () from /lib/x86_64-linux-gnu/libc.so.6 #1 0x00007ffff684326e in ndb_socket_poller::poll_unsafe (this=0x726ca0, timeout=1000) at /home/ushastry/mybuild/mysql-cluster-com-7.3.2/storage/ndb/include/portlib/ndb_socket_poller.h:254 #2 0x00007ffff685e413 in ndb_socket_poller::poll (this=0x726ca0, timeout=1000) at /home/ushastry/mybuild/mysql-cluster-com-7.3.2/storage/ndb/include/portlib/ndb_socket_poller.h:285 #3 0x00007ffff686273e in SocketServer::doAccept (this=0x726c60) at /home/ushastry/mybuild/mysql-cluster-com-7.3.2/storage/ndb/src/common/util/SocketServer.cpp:175 #4 0x00007ffff6862aa2 in SocketServer::doRun (this=0x726c60) at /home/ushastry/mybuild/mysql-cluster-com-7.3.2/storage/ndb/src/common/util/SocketServer.cpp:278 #5 0x00007ffff6862932 in socketServerThread_C (_ss=0x726c60) at /home/ushastry/mybuild/mysql-cluster-com-7.3.2/storage/ndb/src/common/util/SocketServer.cpp:230 #6 0x00007ffff685c481 in ndb_thread_wrapper (_ss=0x7fffd8003000) at /home/ushastry/mybuild/mysql-cluster-com-7.3.2/storage/ndb/src/common/portlib/NdbThread.c:189 #7 0x00007ffff7bc4e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #8 0x00007ffff71aacbd in clone () from /lib/x86_64-linux-gnu/libc.so.6 #9 0x0000000000000000 in ?? () Thread 16 (Thread 0x7fffdf191700 (LWP 18866)): #0 0x00007ffff71a4023 in select () from /lib/x86_64-linux-gnu/libc.so.6 #1 0x00007ffff683c79d in NdbSleep_MilliSleep (milliseconds=100) at /home/ushastry/mybuild/mysql-cluster-com-7.3.2/storage/ndb/include/portlib/NdbSleep.h:80 #2 0x00007ffff6840ae9 in TransporterRegistry::start_clients_thread (this=0x70a640) at /home/ushastry/mybuild/mysql-cluster-com-7.3.2/storage/ndb/src/common/transporter/TransporterRegistry.cpp:1858 #3 0x00007ffff68402b8 in run_start_clients_C (me=0x70a640) ---Type to continue, or q to quit--- at /home/ushastry/mybuild/mysql-cluster-com-7.3.2/storage/ndb/src/common/transporter/TransporterRegistry.cpp:1660 #4 0x00007ffff685c481 in ndb_thread_wrapper (_ss=0x7fffd8002e80) at /home/ushastry/mybuild/mysql-cluster-com-7.3.2/storage/ndb/src/common/portlib/NdbThread.c:189 #5 0x00007ffff7bc4e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #6 0x00007ffff71aacbd in clone () from /lib/x86_64-linux-gnu/libc.so.6 #7 0x0000000000000000 in ?? () Thread 15 (Thread 0x7fffdf1b2700 (LWP 18865)): #0 0x00007ffff71a4023 in select () from /lib/x86_64-linux-gnu/libc.so.6 #1 0x00007ffff689f3d7 in NdbSleep_MilliSleep (milliseconds=20) at /home/ushastry/mybuild/mysql-cluster-com-7.3.2/storage/ndb/include/portlib/NdbSleep.h:80 #2 0x00007ffff68a0492 in ClusterMgr::threadMain (this=0x72cde0) at /home/ushastry/mybuild/mysql-cluster-com-7.3.2/storage/ndb/src/ndbapi/ClusterMgr.cpp:358 #3 0x00007ffff689f3f6 in runClusterMgr_C (me=0x72cde0) at /home/ushastry/mybuild/mysql-cluster-com-7.3.2/storage/ndb/src/ndbapi/ClusterMgr.cpp:52 #4 0x00007ffff685c481 in ndb_thread_wrapper (_ss=0x708660) at /home/ushastry/mybuild/mysql-cluster-com-7.3.2/storage/ndb/src/common/portlib/NdbThread.c:189 #5 0x00007ffff7bc4e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #6 0x00007ffff71aacbd in clone () from /lib/x86_64-linux-gnu/libc.so.6 #7 0x0000000000000000 in ?? () Thread 14 (Thread 0x7fffdf1d3700 (LWP 18864)): #0 0x00007ffff7bc90fe in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 #1 0x00007ffff685d839 in NdbCondition_WaitTimeoutAbs (p_cond=0x6678b0, p_mutex=0x6678f0, abstime=0x7fffdf1d2d90) at /home/ushastry/mybuild/mysql-cluster-com-7.3.2/storage/ndb/src/common/portlib/NdbCondition.c:210 #2 0x00007ffff685d715 in NdbCondition_WaitTimeout (p_cond=0x6678b0, p_mutex=0x6678f0, msecs=1) at /home/ushastry/mybuild/mysql-cluster-com-7.3.2/storage/ndb/src/common/portlib/NdbCondition.c:153 #3 0x00007ffff68292c8 in TransporterFacade::threadMainSend (this=0x726b90) at /home/ushastry/mybuild/mysql-cluster-com-7.3.2/storage/ndb/src/ndbapi/TransporterFacade.cpp:505 #4 0x00007ffff6828ff0 in runSendRequest_C (me=0x726b90) at /home/ushastry/mybuild/mysql-cluster-com-7.3.2/storage/ndb/src/ndbapi/TransporterFacade.cpp:449 #5 0x00007ffff685c481 in ndb_thread_wrapper (_ss=0x6e2af0) at /home/ushastry/mybuild/mysql-cluster-com-7.3.2/storage/ndb/src/common/portlib/NdbThread.c:189 #6 0x00007ffff7bc4e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #7 0x00007ffff71aacbd in clone () from /lib/x86_64-linux-gnu/libc.so.6 #8 0x0000000000000000 in ?? () Thread 13 (Thread 0x7fffdf1f4700 (LWP 18863)): #0 0x00007ffff71a4023 in select () from /lib/x86_64-linux-gnu/libc.so.6 #1 0x00007ffff6828453 in NdbSleep_MilliSleep (milliseconds=100) at /home/ushastry/mybuild/mysql-cluster-com-7.3.2/storage/ndb/include/portlib/NdbSleep.h:80 #2 0x00007ffff6829a7c in TransporterFacade::threadMainReceive (this=0x726b90) at /home/ushastry/mybuild/mysql-cluster-com-7.3.2/storage/ndb/src/ndbapi/TransporterFacade.cpp:757 ---Type to continue, or q to quit--- #3 0x00007ffff68294f9 in runReceiveResponse_C (me=0x726b90) at /home/ushastry/mybuild/mysql-cluster-com-7.3.2/storage/ndb/src/ndbapi/TransporterFacade.cpp:565 #4 0x00007ffff685c481 in ndb_thread_wrapper (_ss=0x712120) at /home/ushastry/mybuild/mysql-cluster-com-7.3.2/storage/ndb/src/common/portlib/NdbThread.c:189 #5 0x00007ffff7bc4e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #6 0x00007ffff71aacbd in clone () from /lib/x86_64-linux-gnu/libc.so.6 #7 0x0000000000000000 in ?? () Thread 12 (Thread 0x7ffff7ed1700 (LWP 18861)): #0 0x00007ffff719f303 in poll () from /lib/x86_64-linux-gnu/libc.so.6 #1 0x00007ffff684326e in ndb_socket_poller::poll_unsafe (this=0x6cd710, timeout=1000) at /home/ushastry/mybuild/mysql-cluster-com-7.3.2/storage/ndb/include/portlib/ndb_socket_poller.h:254 #2 0x00007ffff685e413 in ndb_socket_poller::poll (this=0x6cd710, timeout=1000) at /home/ushastry/mybuild/mysql-cluster-com-7.3.2/storage/ndb/include/portlib/ndb_socket_poller.h:285 #3 0x00007ffff686273e in SocketServer::doAccept (this=0x6cd6d0) at /home/ushastry/mybuild/mysql-cluster-com-7.3.2/storage/ndb/src/common/util/SocketServer.cpp:175 #4 0x00007ffff6862aa2 in SocketServer::doRun (this=0x6cd6d0) at /home/ushastry/mybuild/mysql-cluster-com-7.3.2/storage/ndb/src/common/util/SocketServer.cpp:278 #5 0x00007ffff6862932 in socketServerThread_C (_ss=0x6cd6d0) at /home/ushastry/mybuild/mysql-cluster-com-7.3.2/storage/ndb/src/common/util/SocketServer.cpp:230 #6 0x00007ffff685c481 in ndb_thread_wrapper (_ss=0x7fffd0000b90) at /home/ushastry/mybuild/mysql-cluster-com-7.3.2/storage/ndb/src/common/portlib/NdbThread.c:189 #7 0x00007ffff7bc4e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #8 0x00007ffff71aacbd in clone () from /lib/x86_64-linux-gnu/libc.so.6 #9 0x0000000000000000 in ?? () Thread 11 (Thread 0x7ffff7ef2700 (LWP 18860)): #0 0x00007ffff71a4023 in select () from /lib/x86_64-linux-gnu/libc.so.6 #1 0x00007ffff683c79d in NdbSleep_MilliSleep (milliseconds=100) at /home/ushastry/mybuild/mysql-cluster-com-7.3.2/storage/ndb/include/portlib/NdbSleep.h:80 #2 0x00007ffff6840ae9 in TransporterRegistry::start_clients_thread (this=0x676830) at /home/ushastry/mybuild/mysql-cluster-com-7.3.2/storage/ndb/src/common/transporter/TransporterRegistry.cpp:1858 #3 0x00007ffff68402b8 in run_start_clients_C (me=0x676830) at /home/ushastry/mybuild/mysql-cluster-com-7.3.2/storage/ndb/src/common/transporter/TransporterRegistry.cpp:1660 #4 0x00007ffff685c481 in ndb_thread_wrapper (_ss=0x7fffd0000a10) at /home/ushastry/mybuild/mysql-cluster-com-7.3.2/storage/ndb/src/common/portlib/NdbThread.c:189 #5 0x00007ffff7bc4e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #6 0x00007ffff71aacbd in clone () from /lib/x86_64-linux-gnu/libc.so.6 #7 0x0000000000000000 in ?? () Thread 10 (Thread 0x7ffff7f13700 (LWP 18859)): #0 0x00007ffff71a4023 in select () from /lib/x86_64-linux-gnu/libc.so.6 ---Type to continue, or q to quit--- #1 0x00007ffff689f3d7 in NdbSleep_MilliSleep (milliseconds=20) at /home/ushastry/mybuild/mysql-cluster-com-7.3.2/storage/ndb/include/portlib/NdbSleep.h:80 #2 0x00007ffff68a0492 in ClusterMgr::threadMain (this=0x6fc710) at /home/ushastry/mybuild/mysql-cluster-com-7.3.2/storage/ndb/src/ndbapi/ClusterMgr.cpp:358 #3 0x00007ffff689f3f6 in runClusterMgr_C (me=0x6fc710) at /home/ushastry/mybuild/mysql-cluster-com-7.3.2/storage/ndb/src/ndbapi/ClusterMgr.cpp:52 #4 0x00007ffff685c481 in ndb_thread_wrapper (_ss=0x6369a0) at /home/ushastry/mybuild/mysql-cluster-com-7.3.2/storage/ndb/src/common/portlib/NdbThread.c:189 #5 0x00007ffff7bc4e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #6 0x00007ffff71aacbd in clone () from /lib/x86_64-linux-gnu/libc.so.6 #7 0x0000000000000000 in ?? () Thread 9 (Thread 0x7ffff7f34700 (LWP 18858)): #0 0x00007ffff7bc90fe in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 #1 0x00007ffff685d839 in NdbCondition_WaitTimeoutAbs (p_cond=0x675c80, p_mutex=0x689c70, abstime=0x7ffff7f33d90) at /home/ushastry/mybuild/mysql-cluster-com-7.3.2/storage/ndb/src/common/portlib/NdbCondition.c:210 #2 0x00007ffff685d715 in NdbCondition_WaitTimeout (p_cond=0x675c80, p_mutex=0x689c70, msecs=1) at /home/ushastry/mybuild/mysql-cluster-com-7.3.2/storage/ndb/src/common/portlib/NdbCondition.c:153 #3 0x00007ffff68292c8 in TransporterFacade::threadMainSend (this=0x6cd600) at /home/ushastry/mybuild/mysql-cluster-com-7.3.2/storage/ndb/src/ndbapi/TransporterFacade.cpp:505 #4 0x00007ffff6828ff0 in runSendRequest_C (me=0x6cd600) at /home/ushastry/mybuild/mysql-cluster-com-7.3.2/storage/ndb/src/ndbapi/TransporterFacade.cpp:449 #5 0x00007ffff685c481 in ndb_thread_wrapper (_ss=0x64e910) at /home/ushastry/mybuild/mysql-cluster-com-7.3.2/storage/ndb/src/common/portlib/NdbThread.c:189 #6 0x00007ffff7bc4e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #7 0x00007ffff71aacbd in clone () from /lib/x86_64-linux-gnu/libc.so.6 #8 0x0000000000000000 in ?? () Thread 8 (Thread 0x7ffff7f55700 (LWP 18857)): #0 0x00007ffff71a4023 in select () from /lib/x86_64-linux-gnu/libc.so.6 #1 0x00007ffff6828453 in NdbSleep_MilliSleep (milliseconds=100) at /home/ushastry/mybuild/mysql-cluster-com-7.3.2/storage/ndb/include/portlib/NdbSleep.h:80 #2 0x00007ffff6829a7c in TransporterFacade::threadMainReceive (this=0x6cd600) at /home/ushastry/mybuild/mysql-cluster-com-7.3.2/storage/ndb/src/ndbapi/TransporterFacade.cpp:757 #3 0x00007ffff68294f9 in runReceiveResponse_C (me=0x6cd600) at /home/ushastry/mybuild/mysql-cluster-com-7.3.2/storage/ndb/src/ndbapi/TransporterFacade.cpp:565 #4 0x00007ffff685c481 in ndb_thread_wrapper (_ss=0x64e8c0) at /home/ushastry/mybuild/mysql-cluster-com-7.3.2/storage/ndb/src/common/portlib/NdbThread.c:189 #5 0x00007ffff7bc4e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #6 0x00007ffff71aacbd in clone () from /lib/x86_64-linux-gnu/libc.so.6 #7 0x0000000000000000 in ?? () Thread 7 (Thread 0x7ffff7fa3700 (LWP 18855)): ---Type to continue, or q to quit--- #0 0x00007ffff719f303 in poll () from /lib/x86_64-linux-gnu/libc.so.6 #1 0x00007ffff684326e in ndb_socket_poller::poll_unsafe (this=0x6b6c60, timeout=1000) at /home/ushastry/mybuild/mysql-cluster-com-7.3.2/storage/ndb/include/portlib/ndb_socket_poller.h:254 #2 0x00007ffff685e413 in ndb_socket_poller::poll (this=0x6b6c60, timeout=1000) at /home/ushastry/mybuild/mysql-cluster-com-7.3.2/storage/ndb/include/portlib/ndb_socket_poller.h:285 #3 0x00007ffff686273e in SocketServer::doAccept (this=0x6b6c20) at /home/ushastry/mybuild/mysql-cluster-com-7.3.2/storage/ndb/src/common/util/SocketServer.cpp:175 #4 0x00007ffff6862aa2 in SocketServer::doRun (this=0x6b6c20) at /home/ushastry/mybuild/mysql-cluster-com-7.3.2/storage/ndb/src/common/util/SocketServer.cpp:278 #5 0x00007ffff6862932 in socketServerThread_C (_ss=0x6b6c20) at /home/ushastry/mybuild/mysql-cluster-com-7.3.2/storage/ndb/src/common/util/SocketServer.cpp:230 #6 0x00007ffff685c481 in ndb_thread_wrapper (_ss=0x7fffe8000b90) at /home/ushastry/mybuild/mysql-cluster-com-7.3.2/storage/ndb/src/common/portlib/NdbThread.c:189 #7 0x00007ffff7bc4e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #8 0x00007ffff71aacbd in clone () from /lib/x86_64-linux-gnu/libc.so.6 #9 0x0000000000000000 in ?? () Thread 6 (Thread 0x7ffff7fc4700 (LWP 18854)): #0 0x00007ffff71a4023 in select () from /lib/x86_64-linux-gnu/libc.so.6 #1 0x00007ffff683c79d in NdbSleep_MilliSleep (milliseconds=100) at /home/ushastry/mybuild/mysql-cluster-com-7.3.2/storage/ndb/include/portlib/NdbSleep.h:80 #2 0x00007ffff6840ae9 in TransporterRegistry::start_clients_thread (this=0x690710) at /home/ushastry/mybuild/mysql-cluster-com-7.3.2/storage/ndb/src/common/transporter/TransporterRegistry.cpp:1858 #3 0x00007ffff68402b8 in run_start_clients_C (me=0x690710) at /home/ushastry/mybuild/mysql-cluster-com-7.3.2/storage/ndb/src/common/transporter/TransporterRegistry.cpp:1660 #4 0x00007ffff685c481 in ndb_thread_wrapper (_ss=0x7fffe8000a10) at /home/ushastry/mybuild/mysql-cluster-com-7.3.2/storage/ndb/src/common/portlib/NdbThread.c:189 #5 0x00007ffff7bc4e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #6 0x00007ffff71aacbd in clone () from /lib/x86_64-linux-gnu/libc.so.6 #7 0x0000000000000000 in ?? () Thread 5 (Thread 0x7ffff7fe5700 (LWP 18853)): #0 0x00007ffff71a4023 in select () from /lib/x86_64-linux-gnu/libc.so.6 #1 0x00007ffff689f3d7 in NdbSleep_MilliSleep (milliseconds=20) at /home/ushastry/mybuild/mysql-cluster-com-7.3.2/storage/ndb/include/portlib/NdbSleep.h:80 #2 0x00007ffff68a0492 in ClusterMgr::threadMain (this=0x6bcda0) at /home/ushastry/mybuild/mysql-cluster-com-7.3.2/storage/ndb/src/ndbapi/ClusterMgr.cpp:358 #3 0x00007ffff689f3f6 in runClusterMgr_C (me=0x6bcda0) at /home/ushastry/mybuild/mysql-cluster-com-7.3.2/storage/ndb/src/ndbapi/ClusterMgr.cpp:52 #4 0x00007ffff685c481 in ndb_thread_wrapper (_ss=0x647820) at /home/ushastry/mybuild/mysql-cluster-com-7.3.2/storage/ndb/src/common/portlib/NdbThread.c:189 #5 0x00007ffff7bc4e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #6 0x00007ffff71aacbd in clone () from /lib/x86_64-linux-gnu/libc.so.6 ---Type to continue, or q to quit--- #7 0x0000000000000000 in ?? () Thread 4 (Thread 0x7ffff4b2b700 (LWP 18852)): #0 0x00007ffff7bc90fe in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 #1 0x00007ffff685d839 in NdbCondition_WaitTimeoutAbs (p_cond=0x669c90, p_mutex=0x66fed0, abstime=0x7ffff4b2ad90) at /home/ushastry/mybuild/mysql-cluster-com-7.3.2/storage/ndb/src/common/portlib/NdbCondition.c:210 #2 0x00007ffff685d715 in NdbCondition_WaitTimeout (p_cond=0x669c90, p_mutex=0x66fed0, msecs=10) at /home/ushastry/mybuild/mysql-cluster-com-7.3.2/storage/ndb/src/common/portlib/NdbCondition.c:153 #3 0x00007ffff68292c8 in TransporterFacade::threadMainSend (this=0x6b6b50) at /home/ushastry/mybuild/mysql-cluster-com-7.3.2/storage/ndb/src/ndbapi/TransporterFacade.cpp:505 #4 0x00007ffff6828ff0 in runSendRequest_C (me=0x6b6b50) at /home/ushastry/mybuild/mysql-cluster-com-7.3.2/storage/ndb/src/ndbapi/TransporterFacade.cpp:449 #5 0x00007ffff685c481 in ndb_thread_wrapper (_ss=0x654180) at /home/ushastry/mybuild/mysql-cluster-com-7.3.2/storage/ndb/src/common/portlib/NdbThread.c:189 #6 0x00007ffff7bc4e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #7 0x00007ffff71aacbd in clone () from /lib/x86_64-linux-gnu/libc.so.6 #8 0x0000000000000000 in ?? () Thread 3 (Thread 0x7ffff4b4c700 (LWP 18851)): #0 0x00007ffff71a4023 in select () from /lib/x86_64-linux-gnu/libc.so.6 #1 0x00007ffff6828453 in NdbSleep_MilliSleep (milliseconds=100) at /home/ushastry/mybuild/mysql-cluster-com-7.3.2/storage/ndb/include/portlib/NdbSleep.h:80 #2 0x00007ffff6829a7c in TransporterFacade::threadMainReceive (this=0x6b6b50) at /home/ushastry/mybuild/mysql-cluster-com-7.3.2/storage/ndb/src/ndbapi/TransporterFacade.cpp:757 #3 0x00007ffff68294f9 in runReceiveResponse_C (me=0x6b6b50) at /home/ushastry/mybuild/mysql-cluster-com-7.3.2/storage/ndb/src/ndbapi/TransporterFacade.cpp:565 #4 0x00007ffff685c481 in ndb_thread_wrapper (_ss=0x65cb40) at /home/ushastry/mybuild/mysql-cluster-com-7.3.2/storage/ndb/src/common/portlib/NdbThread.c:189 #5 0x00007ffff7bc4e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #6 0x00007ffff71aacbd in clone () from /lib/x86_64-linux-gnu/libc.so.6 #7 0x0000000000000000 in ?? () Thread 1 (Thread 0x7ffff6ead700 (LWP 18846)): #0 0x00007ffff70ed425 in raise () from /lib/x86_64-linux-gnu/libc.so.6 #1 0x00007ffff70f0b8b in abort () from /lib/x86_64-linux-gnu/libc.so.6 #2 0x00007ffff70e60ee in ?? () from /lib/x86_64-linux-gnu/libc.so.6 #3 0x00007ffff70e6192 in __assert_fail () from /lib/x86_64-linux-gnu/libc.so.6 #4 0x0000000000412c2a in unregister_event (c=0x1180012) at /home/ushastry/mybuild/mysql-cluster-com-7.3.2/storage/ndb/memcache/extra/memcached/daemon/memcached.c:4968 #5 0x00000000004148b2 in event_handler (fd=59, which=2, arg=0x1180012) at /home/ushastry/mybuild/mysql-cluster-com-7.3.2/storage/ndb/memcache/extra/memcached/daemon/memcached.c:5585 #6 0x00007ffff7985bfc in event_process_active_single_queue (activeq=0x62e470, base=0x62e030) at event.c:1350 ---Type to continue, or q to quit--- #7 event_process_active (base=) at event.c:1420 #8 event_base_loop (base=0x62e030, flags=0) at event.c:1621 #9 0x0000000000418ad7 in main (argc=3, argv=0x7fffffffe228) at /home/ushastry/mybuild/mysql-cluster-com-7.3.2/storage/ndb/memcache/extra/memcached/daemon/memcached.c:7535