Bug #29140 ndb_restore cores on restore of disk data, data
Submitted: 15 Jun 2007 14:36 Modified: 25 Jun 2007 6:45
Reporter: Jonathan Miller Email Updates:
Status: Can't repeat Impact on me:
None 
Category:MySQL Cluster: NDB API Severity:S1 (Critical)
Version:mysql-5.1.19 ndb-6.2.3 OS:Linux
Assigned to: Guangbao Ni CPU Architecture:Any

[15 Jun 2007 14:36] Jonathan Miller 
Description:
Hi,

In trying to restore the database from backup with regards to http://bugs.mysql.com/bug.php?id=29127 ndb_restore cores once it get to the restore of actaul data.

Nodeid = 2
Backup Id = 1
backup path = ./BACKUP/BACKUP-1/
Opening file './BACKUP/BACKUP-1/BACKUP-1.2.ctl'
File size 63268 bytes
Ndb version in backup files: ndb-6.2.3
Connected to ndb!!
Creating logfile group: lg1...done
Creating tablespace: ts_subsdata...done
Creating undofile "./lg1/undofile.dat"...done
Creating datafile "./ts_subsdata/datafile2.dat"...done
Creating undofile "./lg1/undofile1.dat"...done
Creating datafile "./ts_subsdata/datafile.dat"...done
Successfully restored table `test/def/HLR_SERV_CENTERS_D`
Successfully restored table event REPL$test/HLR_SERV_CENTERS_D
Successfully restored table `test/def/SUBS_DATA_D`
Successfully restored table event REPL$test/SUBS_DATA_D
Successfully restored table `test/def/HLR_PDP_DATAS_D`
Successfully restored table event REPL$test/HLR_PDP_DATAS_D
Successfully restored table `test/def/SUBS_BS_D`
Successfully restored table event REPL$test/SUBS_BS_D
Successfully restored table `test/def/HLR_SUBSCRIBER_D`
Successfully restored table event REPL$test/HLR_SUBSCRIBER_D
Successfully restored table `test/def/CUG_SUBSCRIPTIONS_D`
Successfully restored table event REPL$test/CUG_SUBSCRIPTIONS_D
Successfully restored table `test/def/HLR_TRACE_DATA_D`
Successfully restored table event REPL$test/HLR_TRACE_DATA_D
Successfully restored table `test/def/SUBS_LCS_D`
Successfully restored table event REPL$test/SUBS_LCS_D
Successfully restored table `test/def/HLR_MULTI_IMSI_D`
Successfully restored table event REPL$test/HLR_MULTI_IMSI_D
Successfully restored table `test/def/SS_SERVS_D`
Successfully restored table event REPL$test/SS_SERVS_D
Successfully restored table `test/def/CAMEL_DATA_D`
Successfully restored table event REPL$test/CAMEL_DATA_D
Successfully created index `PRIMARY` on `SUBS_LCS_D`
Successfully created index `PRIMARY` on `SS_SERVS_D`
Successfully created index `PRIMARY` on `CUG_SUBSCRIPTIONS_D`
Successfully created index `PRIMARY` on `SUBS_BS_D`
Successfully created index `PRIMARY` on `HLR_PDP_DATAS_D`
Successfully created index `PRIMARY` on `SUBS_DATA_D`
Successfully created index `PRIMARY` on `HLR_SUBSCRIBER_D`
Successfully created index `PRIMARY` on `HLR_TRACE_DATA_D`
Successfully created index `PRIMARY` on `HLR_SERV_CENTERS_D`
Successfully created index `PRIMARY` on `HLR_MULTI_IMSI_D`
Successfully created index `PRIMARY` on `CAMEL_DATA_D`
Opening file './BACKUP/BACKUP-1/BACKUP-1-0.2.Data'
File size 2036440680 bytes
_____________________________________________________
Processing data in table: test/def/HLR_SERV_CENTERS_D(17) fragment 0
Segmentation fault (core dumped)

(gdb) bt
#0  0x08060a0b in Twiddle64 (in=624975603367411713) at restore/Restore.cpp:1005
#1  0x0805ea37 in BackupRestore::object (this=0x8aaa6c0, type=3219220272, ptr=0x0)
    at restore/consumer_restore.cpp:589
#2  0x0805b556 in BackupRestore::map_in_frm (this=0x1, new_data=0x8aaa884 "\uffff\uffff\uffff\uffff",
    data=0xbfe18bb0 "\v\uffff\uffff\uffff$\uffff\uffff\uffff/\uffff\uffff\uffff?\uffff\uffff\uffffM\uffff\uffff\uffffm\uffff\uffff\uffff\234\uffff\u1fee\uffff\uffff\uffff\uffff\uffff\uffff\uffff\u036b\uffff\uffff\220\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\v\uffff\uffff\uffff)\uffff\uffff\uffffE\uffff\uffff\uffff\231\uffff\u1fee\uffff\u1ffd\uffff\uffff\uffff\u03ae\uffff\uffff\u076e\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\006\uffff\uffff\uffff6\uffff\uffff\uffffX\uffff\uffff\ufffff\uffff\uffff\uffff{\uffff\u1fe9\uffff\uffff\uffff", data_len=3219229504,
    new_data_len=0xb9b898) at restore/consumer_restore.cpp:400
#3  0x0019dd7f in ?? ()
#4  0x00000001 in ?? ()
#5  0x08aaa884 in ?? ()
#6  0xbfe18bb0 in ?? ()
#7  0xbfe18b40 in ?? ()
#8  0x00b9b898 in ?? ()
#9  0x00ba9878 in ?? ()
#10 0xb7f12670 in ?? ()
#11 0x00000001 in ?? ()
#12 0x002adff4 in ?? ()
#13 0x00ba8ca0 in ?? ()
#14 0x0808db10 in plane03 ()
#15 0xbfe18b58 in ?? ()
#16 0xbfe18b00 in ?? ()
#17 0x0019dd44 in ?? ()
#18 0x00000000 in ?? ()

How to repeat:
Try to restore using the backup files
[20 Jun 2007 9:01] Guangbao Ni 
i do it, ndb_restore can do it successfully.

[root@dev3-221 logfilestatrep]# ./storage/ndb/tools/ndb_restore -n1 -b2 -m -r /tmp/BACKUP/BACKUP-2/ -c localhost:9310
Nodeid = 1
Backup Id = 2
backup path = /tmp/BACKUP/BACKUP-2/
Opening file '/tmp/BACKUP/BACKUP-2/BACKUP-2.1.ctl'
File size 63160 bytes
Ndb version in backup files: ndb-6.2.3
Connected to ndb!!
Creating logfile group: lg1...done
Creating tablespace: ts_subsdata...done
Creating undofile "./lg1/undofile.dat"...done
Creating datafile "./ts_subsdata/datafile2.dat"...done
Creating datafile "./ts_subsdata/datafile.dat"...done
Successfully restored table `test/def/HLR_SERV_CENTERS_D`
Successfully restored table event REPL$test/HLR_SERV_CENTERS_D
Successfully restored table `test/def/SUBS_DATA_D`
Successfully restored table event REPL$test/SUBS_DATA_D
Successfully restored table `test/def/HLR_PDP_DATAS_D`
Successfully restored table event REPL$test/HLR_PDP_DATAS_D
Successfully restored table `test/def/SUBS_BS_D`
Successfully restored table event REPL$test/SUBS_BS_D
Successfully restored table `test/def/HLR_SUBSCRIBER_D`
Successfully restored table event REPL$test/HLR_SUBSCRIBER_D
Successfully restored table `test/def/CUG_SUBSCRIPTIONS_D`
Successfully restored table event REPL$test/CUG_SUBSCRIPTIONS_D
Successfully restored table `test/def/HLR_TRACE_DATA_D`
Successfully restored table event REPL$test/HLR_TRACE_DATA_D
Successfully restored table `test/def/SUBS_LCS_D`
Successfully restored table event REPL$test/SUBS_LCS_D
Successfully restored table `test/def/HLR_MULTI_IMSI_D`
Successfully restored table event REPL$test/HLR_MULTI_IMSI_D
Successfully restored table `test/def/SS_SERVS_D`
Successfully restored table event REPL$test/SS_SERVS_D
Successfully restored table `test/def/CAMEL_DATA_D`
Successfully restored table event REPL$test/CAMEL_DATA_D
Successfully created index `PRIMARY` on `HLR_SERV_CENTERS_D`
Successfully created index `PRIMARY` on `HLR_MULTI_IMSI_D`
Successfully created index `PRIMARY` on `CAMEL_DATA_D`
Successfully created index `PRIMARY` on `SUBS_LCS_D`
Successfully created index `PRIMARY` on `SS_SERVS_D`
Successfully created index `PRIMARY` on `CUG_SUBSCRIPTIONS_D`
Successfully created index `PRIMARY` on `SUBS_BS_D`
Successfully created index `PRIMARY` on `HLR_PDP_DATAS_D`
Successfully created index `PRIMARY` on `SUBS_DATA_D`
Successfully created index `PRIMARY` on `HLR_SUBSCRIBER_D`
Successfully created index `PRIMARY` on `HLR_TRACE_DATA_D`
Opening file '/tmp/BACKUP/BACKUP-2/BACKUP-2-0.1.Data'
File size 821028008 bytes
_____________________________________________________
Processing data in table: test/def/HLR_SERV_CENTERS_D(16) fragment 0
_____________________________________________________
Processing data in table: test/def/SUBS_DATA_D(10) fragment 0
_____________________________________________________
Processing data in table: sys/def/NDB$EVENTS_0(1) fragment 0
_____________________________________________________
Processing data in table: sys/def/SYSTAB_0(0) fragment 0
_____________________________________________________
Processing data in table: test/def/HLR_PDP_DATAS_D(28) fragment 0
_____________________________________________________
Processing data in table: test/def/SUBS_BS_D(26) fragment 0
_____________________________________________________
Processing data in table: mysql/def/NDB$BLOB_2_3(3) fragment 0
_____________________________________________________
Processing data in table: test/def/HLR_SUBSCRIBER_D(12) fragment 0
_____________________________________________________
Processing data in table: test/def/CUG_SUBSCRIPTIONS_D(24) fragment 0
_____________________________________________________
Processing data in table: mysql/def/ndb_schema(2) fragment 0
_____________________________________________________
Processing data in table: test/def/HLR_TRACE_DATA_D(14) fragment 0
_____________________________________________________
Processing data in table: mysql/def/ndb_apply_status(4) fragment 0
_____________________________________________________
Processing data in table: test/def/SUBS_LCS_D(20) fragment 0
_____________________________________________________
Processing data in table: test/def/HLR_MULTI_IMSI_D(18) fragment 0
_____________________________________________________
Processing data in table: test/def/SS_SERVS_D(22) fragment 0
_____________________________________________________
Processing data in table: test/def/CAMEL_DATA_D(30) fragment 0
Opening file '/tmp/BACKUP/BACKUP-2/BACKUP-2.1.log'
File size 44 bytes
Restored 601025 tuples and 0 log entries

NDBT_ProgramExit: 0 - OK
[20 Jun 2007 12:22] Guangbao Ni 
And i test i with the newest source code of mysql-5.1-telco tree, ndb_restore hasn't segment falut error and it can be successful.

ndbdev@ndbmaster:/ndb/gni/mysql-5.1-telco> ./storage/ndb/tools/ndb_restore -n2 -b1 -m -r /bugs/Bug29127/BACKUP/BACKUP-1/
Nodeid = 2
Backup Id = 1
backup path = /bugs/Bug29127/BACKUP/BACKUP-1/
Opening file '/bugs/Bug29127/BACKUP/BACKUP-1/BACKUP-1.2.ctl'
File size 63268 bytes
Ndb version in backup files: ndb-6.2.3
Unable to connect with connect string: nodeid=0,localhost:1186
Retrying every 5 seconds. Attempts left: 12
ndbdev@ndbmaster:/ndb/gni/mysql-5.1-telco> ./storage/ndb/tools/ndb_restore -n2 -b1 -m -r /bugs/Bug29127/BACKUP/BACKUP-1/ -c localhost:9310
Nodeid = 2
Backup Id = 1
backup path = /bugs/Bug29127/BACKUP/BACKUP-1/
Opening file '/bugs/Bug29127/BACKUP/BACKUP-1/BACKUP-1.2.ctl'
File size 63268 bytes
Ndb version in backup files: ndb-6.2.3
Connected to ndb!!
Creating logfile group: lg1...done
Creating tablespace: ts_subsdata...done
Creating undofile "./lg1/undofile.dat"...done
Creating datafile "./ts_subsdata/datafile2.dat"...done
Creating undofile "./lg1/undofile1.dat"...done
Creating datafile "./ts_subsdata/datafile.dat"...done
Successfully restored table `test/def/HLR_SERV_CENTERS_D`
Successfully restored table event REPL$test/HLR_SERV_CENTERS_D
Successfully restored table `test/def/SUBS_DATA_D`
Successfully restored table event REPL$test/SUBS_DATA_D
Successfully restored table `test/def/HLR_PDP_DATAS_D`
Successfully restored table event REPL$test/HLR_PDP_DATAS_D
Successfully restored table `test/def/SUBS_BS_D`
Successfully restored table event REPL$test/SUBS_BS_D
Successfully restored table `test/def/HLR_SUBSCRIBER_D`
Successfully restored table event REPL$test/HLR_SUBSCRIBER_D
Successfully restored table `test/def/CUG_SUBSCRIPTIONS_D`
Successfully restored table event REPL$test/CUG_SUBSCRIPTIONS_D
Successfully restored table `test/def/HLR_TRACE_DATA_D`
Successfully restored table event REPL$test/HLR_TRACE_DATA_D
Successfully restored table `test/def/SUBS_LCS_D`
Successfully restored table event REPL$test/SUBS_LCS_D
Successfully restored table `test/def/HLR_MULTI_IMSI_D`
Successfully restored table event REPL$test/HLR_MULTI_IMSI_D
Successfully restored table `test/def/SS_SERVS_D`
Successfully restored table event REPL$test/SS_SERVS_D
Successfully restored table `test/def/CAMEL_DATA_D`
Successfully restored table event REPL$test/CAMEL_DATA_D
Successfully created index `PRIMARY` on `SUBS_LCS_D`
Successfully created index `PRIMARY` on `SS_SERVS_D`
Successfully created index `PRIMARY` on `CUG_SUBSCRIPTIONS_D`
Successfully created index `PRIMARY` on `SUBS_BS_D`
Successfully created index `PRIMARY` on `HLR_PDP_DATAS_D`
Successfully created index `PRIMARY` on `SUBS_DATA_D`
Successfully created index `PRIMARY` on `HLR_SUBSCRIBER_D`
Successfully created index `PRIMARY` on `HLR_TRACE_DATA_D`
Successfully created index `PRIMARY` on `HLR_SERV_CENTERS_D`
Successfully created index `PRIMARY` on `HLR_MULTI_IMSI_D`
Successfully created index `PRIMARY` on `CAMEL_DATA_D`
Opening file '/bugs/Bug29127/BACKUP/BACKUP-1/BACKUP-1-0.2.Data'
File size 2036440680 bytes
_____________________________________________________
Processing data in table: test/def/HLR_SERV_CENTERS_D(17) fragment 0
Temporary error: 266: Time-out in NDB, probably caused by deadlock
Temporary error: 266: Time-out in NDB, probably caused by deadlock
Temporary error: 266: Time-out in NDB, probably caused by deadlock
Temporary error: 266: Time-out in NDB, probably caused by deadlock
Temporary error: 266: Time-out in NDB, probably caused by deadlock
Temporary error: 266: Time-out in NDB, probably caused by deadlock
Temporary error: 410: REDO log files overloaded, consult online manual (decrease TimeBetweenLocalCheckpoints, and|or increase NoOfFragmentLogFiles)
Temporary error: 410: REDO log files overloaded, consult online manual (decrease TimeBetweenLocalCheckpoints, and|or increase NoOfFragmentLogFiles)
Temporary error: 410: REDO log files overloaded, consult online manual (decrease TimeBetweenLocalCheckpoints, and|or increase NoOfFragmentLogFiles)
Temporary error: 410: REDO log files overloaded, consult online manual (decrease TimeBetweenLocalCheckpoints, and|or increase NoOfFragmentLogFiles)
Temporary error: 410: REDO log files overloaded, consult online manual (decrease TimeBetweenLocalCheckpoints, and|or increase NoOfFragmentLogFiles)
Temporary error: 410: REDO log files overloaded, consult online manual (decrease TimeBetweenLocalCheckpoints, and|or increase NoOfFragmentLogFiles)
Temporary error: 410: REDO log files overloaded, consult online manual (decrease TimeBetweenLocalCheckpoints, and|or increase NoOfFragmentLogFiles)
Temporary error: 410: REDO log files overloaded, consult online manual (decrease TimeBetweenLocalCheckpoints, and|or increase NoOfFragmentLogFiles)
Temporary error: 410: REDO log files overloaded, consult online manual (decrease TimeBetweenLocalCheckpoints, and|or increase NoOfFragmentLogFiles)
Temporary error: 410: REDO log files overloaded, consult online manual (decrease TimeBetweenLocalCheckpoints, and|or increase NoOfFragmentLogFiles)
Temporary error: 410: REDO log files overloaded, consult online manual (decrease TimeBetweenLocalCheckpoints, and|or increase NoOfFragmentLogFiles)
Temporary error: 410: REDO log files overloaded, consult online manual (decrease TimeBetweenLocalCheckpoints, and|or increase NoOfFragmentLogFiles)
Temporary error: 410: REDO log files overloaded, consult online manual (decrease TimeBetweenLocalCheckpoints, and|or increase NoOfFragmentLogFiles)
Temporary error: 410: REDO log files overloaded, consult online manual (decrease TimeBetweenLocalCheckpoints, and|or increase NoOfFragmentLogFiles)
Temporary error: 410: REDO log files overloaded, consult online manual (decrease TimeBetweenLocalCheckpoints, and|or increase NoOfFragmentLogFiles)
Temporary error: 410: REDO log files overloaded, consult online manual (decrease TimeBetweenLocalCheckpoints, and|or increase NoOfFragmentLogFiles)
Temporary error: 410: REDO log files overloaded, consult online manual (decrease TimeBetweenLocalCheckpoints, and|or increase NoOfFragmentLogFiles)
_____________________________________________________
Processing data in table: test/def/SUBS_DATA_D(11) fragment 0
_____________________________________________________
Processing data in table: sys/def/NDB$EVENTS_0(1) fragment 0
_____________________________________________________
Processing data in table: sys/def/SYSTAB_0(0) fragment 0
_____________________________________________________
Processing data in table: test/def/HLR_PDP_DATAS_D(29) fragment 0
[20 Jun 2007 15:50] Jonathan Miller 
Hi,

Are you using the same configuration file that the customer and I used?

Thanks
/Jeb
[21 Jun 2007 0:58] Guangbao Ni 
Hi Jeb,
   I use the same configuration file that the customer and you used in my local machine testing for the first time, ndb_restore has no any problem.
  and then i test it again in ndbmaster uses the default configuration parameter of mysql-test-run.pl, it also ok.

  thank you very much!

/Guangbao Ni
[21 Jun 2007 0:59] Guangbao Ni 
Hi Jeb,
   I use the same configuration file that the customer and you used in my local machine testing for the first time, ndb_restore has no any problem.
  And then i test it again in ndbmaster uses the default configuration parameter of mysql-test-run.pl, i just want to see whether the ndb_restore can restore data correctly, it also is ok.

  thank you very much!

/Guangbao Ni
[21 Jun 2007 1:30] Jonathan Miller 
Hi,

Are you using a 32 bit Linux system?

Thanks!
/Jeb
[21 Jun 2007 7:08] Guangbao Ni 
Hi Jeb,
  yes, both(my local machine and ndbmaster box) are all 32 bit linux system.

  thanks.

/Guangbao Ni