Bug #12549 Mysql Server Crashes after a restoration process
Submitted: 12 Aug 2005 12:05 Modified: 13 Aug 2005 18:22
Reporter: Sumit Kumar Roy Email Updates:
Status: Not a Bug Impact on me:
None 
Category:MySQL Server: Replication Severity:S1 (Critical)
Version:4.0.17 OS:Linux (Linux Kernal 2.24)
Assigned to: CPU Architecture:Any

[12 Aug 2005 12:05] Sumit Kumar Roy
Description:
We are having a system in which two instances of Mysql Server has been set up as Master and Slave Mode. On a typical system we have over 10 million records in each table. We are using innodb as our database engine.

We have developed a utility to take backup from the slave instance. This utility copies the complete slave directory and the binary log files of the master and the slave.

While restoring data from the backup, all the files in the master DB and slave DB directory, except the master.info and relay-info.log file in the Slave DB directory, are being replaced by the corresponding files in the backup. 

After completion of restoration, the replication between the Master and the Slave is started explicitely by giving "Change Master to ...".

After restoration of data the Master and the Slave DB keep on crashing whenever we try to access the tables on the system.

We did a backtrace on one such crash. The result is below:
[root@test]# resolve_stack_dump -s /usr/lib/mysql/mysqld.sym -n mysql.stack
0x8071fb4 handle_segfault + 420
0x8292c58 pthread_sighandler + 184
0x82bc897 memcpy + 39
0x81416c4 row_sel_store_mysql_rec + 1428
0x8145f0f row_search_for_mysql + 11471
0x80ceca7 general_fetch__11ha_innobasePcUiUi + 359
0x80ced53 index_next__11ha_innobasePc + 35
0x80c1f36 get_next__12QUICK_SELECT + 86
0x80c375d rr_quick__FP14st_read_record + 29
0x809f189 sub_select__FP4JOINP13st_join_tableb + 345
0x809ee23 do_select__FP4JOINPt4List1Z4ItemP8st_tableP9Procedure + 403
0x8097398 mysql_select__FP3THDP13st_table_listRt4List1Z4ItemP4ItemP8st_orderT4T3T4UlP13select_result + 7000
0x8095806 handle_select__FP3THDP6st_lexP13select_result + 102
0x807ccda mysql_execute_command__Fv + 986
0x8080905 mysql_parse__FP3THDPcUi + 149
0x807be43 dispatch_command__F19enum_server_commandP3THDPcUi + 1443
0x807b88e do_command__FP3THD + 158
0x807b078 handle_one_connection + 648
0x829040c pthread_start_thread + 220
0x82c3b3a thread_start + 4

How to repeat:
Start a master & slave setup. Create tables and insert data.
Backup Process:
1. Synchronize slave with the master.
2. Stop Replication giving "stop slave" to the slave.
3. Flush the slave with read lock
4. Copy all the files including the subdirectories of the Slave DB directory to backup directory.
5. Copy the binary log files of the master and the slave to the backup directory

Restoration Process:
1. Stop the master and the Slave.
2. Replace all the files in the master and the slave directory with that of the backuped files.
3 Donot copy the master.info and relay-info.log files.
4 Bring up the master and the slave DB
5 Start the replication using the "change master to.." getting the log file and position from the master.
6. Try to do some I/O. to the master.
[13 Aug 2005 18:22] Alexander Keremidarski
Thank you for taking the time to report a problem.  Unfortunately
you are not using a current version of the product your reported a
problem with -- the problem might already be fixed. Please download
a new version from http://www.mysql.com/downloads/

If you are able to reproduce the bug with one of the latest versions,
please change the version on this bug report to the version you
tested and change the status back to "Open".  Again, thank you for
your continued support of MySQL.
[16 Aug 2005 3:34] Sumit Kumar Roy
Hi Alexander,
We have been authorised to work on theversion of mysql mentioned.
Regards
Sumit Kumar Roy