Description:
Stress testing done on 2 replicas yesterday
produced the following:
060418 2:51:56 [ERROR] NDB create db: distributing ./atae/ timed out.
Ignoring...
*** glibc detected *** /home/ndbdev/jmiller/builds/libexec/mysqld: corrupted
double-linked list: 0x002378b8 ***
======= Backtrace: =========
/lib/libc.so.6[0x173b71]
/lib/libc.so.6[0x174d8d]
/lib/libc.so.6(malloc+0x74)[0x176492]
/home/ndbdev/jmiller/builds/libexec/mysqld(_Znaj+0x26)[0x8420f9a]
/home/ndbdev/jmiller/builds/libexec/mysqld(_ZN6VectorI13Gci_containerE9push_back
ERKS0_+0x53)[0x8368aeb]
/home/ndbdev/jmiller/builds/libexec/mysqld(_ZN6VectorI13Gci_containerE4fillEjRS0
_+0x23)[0x8368c75]
/home/ndbdev/jmiller/builds/libexec/mysqld[0x8366a6b]
/home/ndbdev/jmiller/builds/libexec/mysqld(_Z11find_bucketP6VectorI13Gci_contain
erEy+0x8c)[0x8368d14]
/home/ndbdev/jmiller/builds/libexec/mysqld(_ZN14NdbEventBuffer24execSUB_GCP_COMP
LETE_REPEPK17SubGcpCompleteRep+0x5b)[0x8366fb1]
/home/ndbdev/jmiller/builds/libexec/mysqld(_ZN3Ndb20handleReceivedSignalEP12NdbA
piSignalP16LinearSectionPtr+0xede)[0x83454cc]
/home/ndbdev/jmiller/builds/libexec/mysqld(_ZN3Ndb14executeMessageEPvP12NdbApiSi
gnalP16LinearSectionPtr+0x2a)[0x8345ba6]
/home/ndbdev/jmiller/builds/libexec/mysqld(_ZN17TransporterFacade8for_eachEP12Nd
bApiSignalP16LinearSectionPtr+0x87)[0x836c00b]
/home/ndbdev/jmiller/builds/libexec/mysqld(_Z7executePvP12SignalHeaderhPjP16Line
arSectionPtr+0x4ae)[0x836d072]
/home/ndbdev/jmiller/builds/libexec/mysqld(_ZN19TransporterRegistry6unpackEPjjt7
IOState+0x28c)[0x83c40c8]
/home/ndbdev/jmiller/builds/libexec/mysqld(_ZN19TransporterRegistry14performRece
iveEv+0x144)[0x839307c]
/home/ndbdev/jmiller/builds/libexec/mysqld(_ZN17TransporterFacade17threadMainRec
eiveEv+0xe0)[0x836b620]
/home/ndbdev/jmiller/builds/libexec/mysqld(runReceiveResponse_C+0x1f)[0x836b6d1]
/home/ndbdev/jmiller/builds/libexec/mysqld[0x83b005c]
/lib/libpthread.so.0[0x684b80]
/lib/libc.so.6(__clone+0x5e)[0x1db9ce]
0x81d0974 handle_segfault + 438
0x629420 (?)
(nil)
0x16e22a (?)
0x173b71 (?)
0x174d8d (?)
0x176492 (?)
0x8420f9a _Znaj + 38
0x8368aeb _ZN6VectorI13Gci_containerE9push_backERKS0_ + 83
0x8368c75 _ZN6VectorI13Gci_containerE4fillEjRS0_ + 35
0x8366a6b _Z19find_bucket_chainedP6VectorI13Gci_containerEy + 153
0x8368d14 _Z11find_bucketP6VectorI13Gci_containerEy + 140
0x8366fb1 _ZN14NdbEventBuffer24execSUB_GCP_COMPLETE_REPEPK17SubGcpCompleteRep +
91
0x83454cc _ZN3Ndb20handleReceivedSignalEP12NdbApiSignalP16LinearSectionPtr +
3806
0x8345ba6 _ZN3Ndb14executeMessageEPvP12NdbApiSignalP16LinearSectionPtr + 42
0x836c00b _ZN17TransporterFacade8for_eachEP12NdbApiSignalP16LinearSectionPtr +
135
0x836d072 _Z7executePvP12SignalHeaderhPjP16LinearSectionPtr + 1198
0x83c40c8 _ZN19TransporterRegistry6unpackEPjjt7IOState + 652
0x839307c _ZN19TransporterRegistry14performReceiveEv + 324
0x836b620 _ZN17TransporterFacade17threadMainReceiveEv + 224
0x836b6d1 runReceiveResponse_C + 31
0x83b005c ndb_thread_wrapper + 104
0x684b80 (?)
0x1db9ce (?)
[18 Apr 20:18] Jonathan Miller
Another one that is close in time of the other that I posted.
060418 2:51:46 [ERROR] NDB: CREATE DATABASE atae: error Can't create database
'atae'; database exists 1007 1 1
*** glibc detected *** /home/ndbdev/jmiller/builds/libexec/mysqld: double free
or corruption (fasttop): 0xb45503c0 ***
======= Backtrace: =========
/lib/libc.so.6[0x5f3124]
/lib/libc.so.6(__libc_free+0x77)[0x5f365f]
/home/ndbdev/jmiller/builds/libexec/mysqld(_ZdaPv+0x17)[0x8420fd3]
/home/ndbdev/jmiller/builds/libexec/mysqld(_ZN17EventBufData_listD1Ev+0x36)[0x83
6836a]
/home/ndbdev/jmiller/builds/libexec/mysqld(_ZN13Gci_containerD1Ev+0x22)[0x836864
2]
/home/ndbdev/jmiller/builds/libexec/mysqld(_ZN6VectorI13Gci_containerE9push_back
ERKS0_+0x13a)[0x8368bd2]
/home/ndbdev/jmiller/builds/libexec/mysqld(_ZN6VectorI13Gci_containerE4fillEjRS0
_+0x23)[0x8368c75]
/home/ndbdev/jmiller/builds/libexec/mysqld[0x8366a6b]
/home/ndbdev/jmiller/builds/libexec/mysqld(_Z11find_bucketP6VectorI13Gci_contain
erEy+0x8c)[0x8368d14]
/home/ndbdev/jmiller/builds/libexec/mysqld(_ZN14NdbEventBuffer11insertDataLEP21N
dbEventOperationImplPK12SubTableDataP16LinearSectionPtr+0x59)[0x8366bd9]
/home/ndbdev/jmiller/builds/libexec/mysqld(_ZN3Ndb20handleReceivedSignalEP12NdbA
piSignalP16LinearSectionPtr+0xfcf)[0x83455bd]
/home/ndbdev/jmiller/builds/libexec/mysqld(_ZN3Ndb14executeMessageEPvP12NdbApiSi
gnalP16LinearSectionPtr+0x2a)[0x8345ba6]
/home/ndbdev/jmiller/builds/libexec/mysqld(_Z7executePvP12SignalHeaderhPjP16Line
arSectionPtr+0xc9)[0x836cc8d]
/home/ndbdev/jmiller/builds/libexec/mysqld(_ZN19TransporterRegistry6unpackEPjjt7
IOState+0x28c)[0x83c40c8]
/home/ndbdev/jmiller/builds/libexec/mysqld(_ZN19TransporterRegistry14performRece
iveEv+0x144)[0x839307c]
/home/ndbdev/jmiller/builds/libexec/mysqld(_ZN17TransporterFacade13external_poll
Ej+0x70)[0x836b74e]
/home/ndbdev/jmiller/builds/libexec/mysqld(_ZN9PollGuard14wait_for_inputEi+0xeb)
[0x836db17]
/home/ndbdev/jmiller/builds/libexec/mysqld(_ZN3Ndb25waitCompletedTransactionsEii
P9PollGuard+0x6c)[0x83442fc]
/home/ndbdev/jmiller/builds/libexec/mysqld(_ZN3Ndb10poll_transEiiP9PollGuard+0x5
6)[0x834440e]
/home/ndbdev/jmiller/builds/libexec/mysqld(_ZN3Ndb11sendPollNdbEiii+0x66)[0x8344
4d8]
/home/ndbdev/jmiller/builds/libexec/mysqld(_ZN14NdbTransaction14executeNoBlobsEN
S_8ExecTypeENS_11AbortOptionEi+0x78)[0x8350dc2]
/home/ndbdev/jmiller/builds/libexec/mysqld(_ZN14NdbTransaction7executeENS_8ExecT
ypeENS_11AbortOptionEi+0x49)[0x8350ea1]
/home/ndbdev/jmiller/builds/libexec/mysqld(_Z20execute_no_commit_ieP13ha_ndbclus
terP14NdbTransaction+0x1f)[0x833305f]
/home/ndbdev/jmiller/builds/libexec/mysqld(_ZN13ha_ndbcluster22read_multi_range_
firstEPP18st_key_multi_rangeS1_jbP17st_handler_buffer+0x77f)[0x832df8d]
/home/ndbdev/jmiller/builds/libexec/mysqld(_ZN18QUICK_RANGE_SELECT8get_nextEv+0x
219)[0x8291f07]
/home/ndbdev/jmiller/builds/libexec/mysqld[0x829b452]
/home/ndbdev/jmiller/builds/libexec/mysqld(_Z12mysql_updateP3THDP13st_table_list
R4ListI4ItemES6_PS4_jP8st_orderm15enum_duplicatesb+0x12c2)[0x8253bcc]
/home/ndbdev/jmiller/builds/libexec/mysqld(_Z21mysql_execute_commandP3THD+0x244b
)[0x81ebb5f]
/home/ndbdev/jmiller/builds/libexec/mysqld(_ZN13sp_instr_stmt9exec_coreEP3THDPj+
0x11)[0x830712f]
/home/ndbdev/jmiller/builds/libexec/mysqld(_ZN13sp_lex_keeper23reset_lex_and_exe
c_coreEP3THDPjbP8sp_instr+0x128)[0x8306f52]
/home/ndbdev/jmiller/builds/libexec/mysqld(_ZN13sp_instr_stmt7executeEP3THDPj+0x
111)[0x8309665]
/home/ndbdev/jmiller/builds/libexec/mysqld(_ZN7sp_head7executeEP3THD+0x2a7)[0x83
04931]
/home/ndbdev/jmiller/builds/libexec/mysqld(_ZN7sp_head17execute_procedureEP3THDP
4ListI4ItemE+0x49c)[0x83057c8]
/home/ndbdev/jmiller/builds/libexec/mysqld(_Z21mysql_execute_commandP3THD+0x628a
)[0x81ef99e]
/home/ndbdev/jmiller/builds/libexec/mysqld(_Z11mysql_parseP3THDPcj+0x217)[0x81f1
6e1]
/home/ndbdev/jmiller/builds/libexec/mysqld(_Z16dispatch_command19enum_server_com
mandP3THDPcj+0x746)[0x81f1f3e]
/home/ndbdev/jmiller/builds/libexec/mysqld(_Z10do_commandP3THD+0x104)[0x81f3080]
/home/ndbdev/jmiller/builds/libexec/mysqld(handle_one_connection+0x2d5)[0x81f343
7]
/lib/libpthread.so.0[0x702b80]
/lib/libc.so.6(__clone+0x5e)[0x65a9ce]
Cluster log at this time:
2006-04-18 03:02:51 [MgmSrvr] INFO -- Node 8: Event buffer status: used=70KB(100%)
alloc=70KB(0%) max=0B apply_gci=1852 latest_gci=13064
How to repeat:
I was currently running the following tests in a 6 host cluster:
1 - cid_ndb_dd.pl - using memory tables - no disk data
4 - tpcb.pl - using mixed data (memory and disk)
1 - DBT2 - using 2 connection with 10 threads each - memory tables
6 - aysnc API programs using memory tables.
Cluster Config:
Host:07 - MySQLD & ndb_mgmd
Host:08 - MySQLD
Host:09 - MySQLD
Host:12 - MySQLD
Host:13 - NDBD & NDBD
Host:14 - NDBD & NDBD
NoOfReplicas: 2
IndexMemory: 500M
DataMemory: 1000M
BackupMemory: 64M
MaxNoOfLocalOperations: 30000
MaxNoOfTables: 100
StopOnError: 1
MaxNoOfConcurrentScans: 100
DataDir: /space/run
FileSystemPath: /space/run
DiskPageBufferMemory: 300M
#DiskPageBufferMemory: 4M
MaxNoOfConcurrentOperations: 30000
NoOfFragmentLogFiles: 50
