MySQL Bugs: #33040: ndb_restore crashes with --print

Bug #33040	ndb_restore crashes with --print_log
Submitted:	6 Dec 2007 17:31	Modified:	14 Sep 12:22
Reporter:	Oli Sennhauser
Status:	Closed
Category:	Server: Cluster	Severity:	S3 (Non-critical)
Version:	mysql-5.1-telco-6.3	OS:	Linux (x86_64 only, 32bit works fine)
Assigned to:	Martin Skold	Target Version:
Triage:	Triaged: D3 (Medium) / R2 (Low) / E2 (Low)

Description:
ndb_restore gives a dump

How to repeat:
ndb_restore --print_log -b 1 -n 14 | tail -n 100
*** glibc detected *** ndb_restore: double free or corruption (!prev): 0x00000000007d2660
***
======= Backtrace: =========
/lib64/libc.so.6[0x2ac7be7a834e]
/lib64/libc.so.6(__libc_free+0x6c)[0x2ac7be7a995c]
ndb_restore(_ZN10NdbRecAttrD1Ev+0x1a)[0x48823a]
ndb_restore(_ZlsR6NdbOutRK10AttributeS+0x95)[0x456805]
ndb_restore(_ZlsR6NdbOutRK8LogEntry+0xd3)[0x457713]
ndb_restore(_ZN13BackupPrinter8logEntryERK8LogEntry+0x21)[0x4563e1]
ndb_restore(main+0x103c)[0x4500dc]
/lib64/libc.so.6(__libc_start_main+0xf4)[0x2ac7be75a154]
ndb_restore[0x44e3d9]
======= Memory map: ========
00400000-0061a000 r-xp 00000000 fd:04 38994                             
/usr/local/mysql/bin/ndb_restore
00719000-0075f000 rw-p 00219000 fd:04 38994                             
/usr/local/mysql/bin/ndb_restore
0075f000-00804000 rw-p 0075f000 00:00 0                                  [heap]
40000000-40001000 ---p 40000000 00:00 0
40001000-40801000 rw-p 40001000 00:00 0
2aaaaaaab000-2aaaaaaae000 rw-p 2aaaaaaab000 00:00 0
2aaaaaab8000-2aaaaaac5000 r-xp 00000000 68:01 102594                    
/lib64/libgcc_s.so.1
2aaaaaac5000-2aaaaabc4000 ---p 0000d000 68:01 102594                    
/lib64/libgcc_s.so.1
2aaaaabc4000-2aaaaabc5000 rw-p 0000c000 68:01 102594                    
/lib64/libgcc_s.so.1
2ac7be165000-2ac7be180000 r-xp 00000000 68:01 102373                    
/lib64/ld-2.4.so
2ac7be180000-2ac7be181000 rw-p 2ac7be180000 00:00 0
2ac7be18c000-2ac7be18d000 rw-p 2ac7be18c000 00:00 0
2ac7be27f000-2ac7be281000 rw-p 0001a000 68:01 102373                    
/lib64/ld-2.4.so
2ac7be281000-2ac7be293000 r-xp 00000000 68:01 102404                    
/lib64/libpthread-2.4.so
2ac7be293000-2ac7be392000 ---p 00012000 68:01 102404                    
/lib64/libpthread-2.4.so
2ac7be392000-2ac7be394000 rw-p 00011000 68:01 102404                    
/lib64/libpthread-2.4.so
2ac7be394000-2ac7be398000 rw-p 2ac7be394000 00:00 0
2ac7be398000-2ac7be3a1000 r-xp 00000000 68:01 102382                    
/lib64/libcrypt-2.4.so
2ac7be3a1000-2ac7be4a0000 ---p 00009000 68:01 102382                    
/lib64/libcrypt-2.4.so
2ac7be4a0000-2ac7be4a3000 rw-p 00008000 68:01 102382                    
/lib64/libcrypt-2.4.so
2ac7be4a3000-2ac7be4d1000 rw-p 2ac7be4a3000 00:00 0
2ac7be4d1000-2ac7be4e4000 r-xp 00000000 68:01 102389                    
/lib64/libnsl-2.4.so
2ac7be4e4000-2ac7be5e3000 ---p 00013000 68:01 102389                    
/lib64/libnsl-2.4.so
2ac7be5e3000-2ac7be5e5000 rw-p 00012000 68:01 102389                    
/lib64/libnsl-2.4.so
2ac7be5e5000-2ac7be5e8000 rw-p 2ac7be5e5000 00:00 0
2ac7be5e8000-2ac7be63c000 r-xp 00000000 68:01 102397                    
/lib64/libm-2.4.so
2ac7be63c000-2ac7be73b000 ---p 00054000 68:01 102397                    
/lib64/libm-2.4.so
2ac7be73b000-2ac7be73d000 rw-p 00053000 68:01 102397                    
/lib64/libm-2.4.so
2ac7be73d000-2ac7be864000 r-xp 00000000 68:01 102378                    
/lib64/libc-2.4.so
2ac7be864000-2ac7be964000 ---p 00127000 68:01 102378                    
/lib64/libc-2.4.so
2ac7be964000-2ac7be967000 r--p 00127000 68:01 102378                    
/lib64/libc-2.4.so
2ac7be967000-2ac7be969000 rw-p 0012a000 68:01 102378                    
/lib64/libc-2.4.so
2ac7be969000-2ac7be970000 rw-p 2ac7be969000 00:00 0
7fffec92f000-7fffec945000 rw-p 7fffec92f000 00:00 0                      [stack]
ffffffffff600000-ffffffffffe00000 ---p 00000000 00:00 0                  [vdso]
Backup Id = 1
Nodeid = 14
backup path = ./
Opening file './BACKUP-1.14.ctl'
File size 136276 bytes
Ndb version in backup files: ndb-6.3.6
Opening file './BACKUP-1.14.log'
File size 15376 bytes
INSERT ser/def/refIDMapping refid=7-udi13.969226

Suggested fix:
no idea

Can't reproduce, can you provide the backup files this happened with?
The behavior probably depends on the backup contents ...

Files are uploaded long time ago...

Works fine on 32bit, crashes on AMD64:

gdb backtrace:

(gdb) bt
#0  0x0000000000489b04 in ndbrecattr_print_formatted (out=@0x77f530, r=@0x7fbfff8e70,
f=@0x77d5e0)
    at ../../../../storage/ndb/include/ndbapi/NdbRecAttr.hpp:346
#1  0x000000000045c7d1 in operator<< (ndbout=@0x77f530, attr=Variable "attr" is not
available.
) at restore/Restore.cpp:1367
#2  0x000000000045ca0a in operator<< (ndbout=@0x77f530, logE=@0x7fbfffb008) at
restore/Restore.cpp:1412
#3  0x00000000004590a7 in BackupPrinter::logEntry (this=0x785d80, logE=Variable "logE" is
not available.
)
    at ../../../storage/ndb/include/util/NdbOut.hpp:87
#4  0x0000000000452de1 in main (argc=Variable "argc" is not available.
) at restore/restore_main.cpp:982

I did test using the ctl and log file uploaded, but didn't reproduct the problem.

Can you upload the config.ini file and all backup files.

Or you can upload SQL files and i can do a backup then restore it.

The process of restore seems ctl file have no problem:

 ./storage/ndb/tools/ndb_restore -c localhost:9310 --print_log -b 1 -n 15 -m -r
/home/zhl/
Backup Id = 1
Nodeid = 15
backup path = /home/zhl/
Opening file '/home/zhl/BACKUP-1.15.ctl'
Ndb version in backup files: ndb-6.3.6
Connected to ndb!!
Successfully restored table `ser/def/if_settings`
Successfully restored table event REPL$ser/if_settings
..........................
..........................
Successfully created index `PRIMARY` on `whitelist_in`
Successfully created index `inid_access` on `voip_instance_user`
Opening file '/home/zhl/BACKUP-1-0.15.Data'
Progress reporting degraded output since fstat failed,errno: 9
Failed to read header of data file. Exiting...

NDBT_ProgramExit: 1 - Failed

Aborted (core dumped)

reverified with 6.3.14

Please try the patch of Bug #40428.

That patch does indeed seem to fix the problem :)

5.1.22-ndb-6.3.6

See bug#40428

Documented fix and closed; see BUG#40428 for changelog entry.