Bug #33040 ndb_restore crashes with --print_log
Submitted: 6 Dec 2007 16:31 Modified: 14 Sep 2009 10:22
Reporter: Oli Sennhauser Email Updates:
Status: Closed Impact on me:
None 
Category:MySQL Cluster: Cluster (NDB) storage engine Severity:S3 (Non-critical)
Version:mysql-5.1-telco-6.3 OS:Linux (x86_64 only, 32bit works fine)
Assigned to: Martin Skold CPU Architecture:Any

[6 Dec 2007 16:31] Oli Sennhauser
Description:
ndb_restore gives a dump

How to repeat:
ndb_restore --print_log -b 1 -n 14 | tail -n 100
*** glibc detected *** ndb_restore: double free or corruption (!prev): 0x00000000007d2660 ***
======= Backtrace: =========
/lib64/libc.so.6[0x2ac7be7a834e]
/lib64/libc.so.6(__libc_free+0x6c)[0x2ac7be7a995c]
ndb_restore(_ZN10NdbRecAttrD1Ev+0x1a)[0x48823a]
ndb_restore(_ZlsR6NdbOutRK10AttributeS+0x95)[0x456805]
ndb_restore(_ZlsR6NdbOutRK8LogEntry+0xd3)[0x457713]
ndb_restore(_ZN13BackupPrinter8logEntryERK8LogEntry+0x21)[0x4563e1]
ndb_restore(main+0x103c)[0x4500dc]
/lib64/libc.so.6(__libc_start_main+0xf4)[0x2ac7be75a154]
ndb_restore[0x44e3d9]
======= Memory map: ========
00400000-0061a000 r-xp 00000000 fd:04 38994                              /usr/local/mysql/bin/ndb_restore
00719000-0075f000 rw-p 00219000 fd:04 38994                              /usr/local/mysql/bin/ndb_restore
0075f000-00804000 rw-p 0075f000 00:00 0                                  [heap]
40000000-40001000 ---p 40000000 00:00 0
40001000-40801000 rw-p 40001000 00:00 0
2aaaaaaab000-2aaaaaaae000 rw-p 2aaaaaaab000 00:00 0
2aaaaaab8000-2aaaaaac5000 r-xp 00000000 68:01 102594                     /lib64/libgcc_s.so.1
2aaaaaac5000-2aaaaabc4000 ---p 0000d000 68:01 102594                     /lib64/libgcc_s.so.1
2aaaaabc4000-2aaaaabc5000 rw-p 0000c000 68:01 102594                     /lib64/libgcc_s.so.1
2ac7be165000-2ac7be180000 r-xp 00000000 68:01 102373                     /lib64/ld-2.4.so
2ac7be180000-2ac7be181000 rw-p 2ac7be180000 00:00 0
2ac7be18c000-2ac7be18d000 rw-p 2ac7be18c000 00:00 0
2ac7be27f000-2ac7be281000 rw-p 0001a000 68:01 102373                     /lib64/ld-2.4.so
2ac7be281000-2ac7be293000 r-xp 00000000 68:01 102404                     /lib64/libpthread-2.4.so
2ac7be293000-2ac7be392000 ---p 00012000 68:01 102404                     /lib64/libpthread-2.4.so
2ac7be392000-2ac7be394000 rw-p 00011000 68:01 102404                     /lib64/libpthread-2.4.so
2ac7be394000-2ac7be398000 rw-p 2ac7be394000 00:00 0
2ac7be398000-2ac7be3a1000 r-xp 00000000 68:01 102382                     /lib64/libcrypt-2.4.so
2ac7be3a1000-2ac7be4a0000 ---p 00009000 68:01 102382                     /lib64/libcrypt-2.4.so
2ac7be4a0000-2ac7be4a3000 rw-p 00008000 68:01 102382                     /lib64/libcrypt-2.4.so
2ac7be4a3000-2ac7be4d1000 rw-p 2ac7be4a3000 00:00 0
2ac7be4d1000-2ac7be4e4000 r-xp 00000000 68:01 102389                     /lib64/libnsl-2.4.so
2ac7be4e4000-2ac7be5e3000 ---p 00013000 68:01 102389                     /lib64/libnsl-2.4.so
2ac7be5e3000-2ac7be5e5000 rw-p 00012000 68:01 102389                     /lib64/libnsl-2.4.so
2ac7be5e5000-2ac7be5e8000 rw-p 2ac7be5e5000 00:00 0
2ac7be5e8000-2ac7be63c000 r-xp 00000000 68:01 102397                     /lib64/libm-2.4.so
2ac7be63c000-2ac7be73b000 ---p 00054000 68:01 102397                     /lib64/libm-2.4.so
2ac7be73b000-2ac7be73d000 rw-p 00053000 68:01 102397                     /lib64/libm-2.4.so
2ac7be73d000-2ac7be864000 r-xp 00000000 68:01 102378                     /lib64/libc-2.4.so
2ac7be864000-2ac7be964000 ---p 00127000 68:01 102378                     /lib64/libc-2.4.so
2ac7be964000-2ac7be967000 r--p 00127000 68:01 102378                     /lib64/libc-2.4.so
2ac7be967000-2ac7be969000 rw-p 0012a000 68:01 102378                     /lib64/libc-2.4.so
2ac7be969000-2ac7be970000 rw-p 2ac7be969000 00:00 0
7fffec92f000-7fffec945000 rw-p 7fffec92f000 00:00 0                      [stack]
ffffffffff600000-ffffffffffe00000 ---p 00000000 00:00 0                  [vdso]
Backup Id = 1
Nodeid = 14
backup path = ./
Opening file './BACKUP-1.14.ctl'
File size 136276 bytes
Ndb version in backup files: ndb-6.3.6
Opening file './BACKUP-1.14.log'
File size 15376 bytes
INSERT ser/def/refIDMapping refid=7-udi13.969226

Suggested fix:
no idea
[6 Dec 2007 17:14] Hartmut Holzgraefe
Can't reproduce, can you provide the backup files this happened with?
The behavior probably depends on the backup contents ...
[19 Dec 2007 13:39] Oli Sennhauser
Files are uploaded long time ago...
[20 Dec 2007 11:10] Hartmut Holzgraefe
Works fine on 32bit, crashes on AMD64:

gdb backtrace:

(gdb) bt
#0  0x0000000000489b04 in ndbrecattr_print_formatted (out=@0x77f530, r=@0x7fbfff8e70, f=@0x77d5e0)
    at ../../../../storage/ndb/include/ndbapi/NdbRecAttr.hpp:346
#1  0x000000000045c7d1 in operator<< (ndbout=@0x77f530, attr=Variable "attr" is not available.
) at restore/Restore.cpp:1367
#2  0x000000000045ca0a in operator<< (ndbout=@0x77f530, logE=@0x7fbfffb008) at restore/Restore.cpp:1412
#3  0x00000000004590a7 in BackupPrinter::logEntry (this=0x785d80, logE=Variable "logE" is not available.
)
    at ../../../storage/ndb/include/util/NdbOut.hpp:87
#4  0x0000000000452de1 in main (argc=Variable "argc" is not available.
) at restore/restore_main.cpp:982
[9 Jan 2008 2:21] li zhou
I did test using the ctl and log file uploaded, but didn't reproduct the problem.

Can you upload the config.ini file and all backup files.

Or you can upload SQL files and i can do a backup then restore it.

The process of restore seems ctl file have no problem:

 ./storage/ndb/tools/ndb_restore -c localhost:9310 --print_log -b 1 -n 15 -m -r /home/zhl/
Backup Id = 1
Nodeid = 15
backup path = /home/zhl/
Opening file '/home/zhl/BACKUP-1.15.ctl'
Ndb version in backup files: ndb-6.3.6
Connected to ndb!!
Successfully restored table `ser/def/if_settings`
Successfully restored table event REPL$ser/if_settings
..........................
..........................
Successfully created index `PRIMARY` on `whitelist_in`
Successfully created index `inid_access` on `voip_instance_user`
Opening file '/home/zhl/BACKUP-1-0.15.Data'
Progress reporting degraded output since fstat failed,errno: 9
Failed to read header of data file. Exiting...

NDBT_ProgramExit: 1 - Failed

Aborted (core dumped)
[10 Jun 2008 16:14] Hartmut Holzgraefe
reverified with 6.3.14
[30 Oct 2008 14:50] li zhou
Please try the patch of Bug #40428.
[16 Jan 2009 9:43] Hartmut Holzgraefe
That patch does indeed seem to fix the problem :)
[30 Apr 2009 13:23] Jonathan Miller
5.1.22-ndb-6.3.6
[10 Sep 2009 7:19] Martin Skold
See bug#40428
[14 Sep 2009 10:22] Jon Stephens
Documented fix and closed; see BUG#40428 for changelog entry.