MySQL Bugs: #823: Replication

Bug #823	Replication
Submitted:	10 Jul 2003 1:02	Modified:	11 Jul 2003 6:34
Reporter:	Anirudh	Email Updates:
Status:	Not a Bug	Impact on me:	None
Category:	MySQL Server	Severity:	S1 (Critical)
Version:	4.0.12	OS:	Linux (Linux Mandrake(9.0))
Assigned to:	Guilhem Bichot	CPU Architecture:	Any

Description:
Server Version : 4.0.12-log
Protocol Version : 10

Platform : Linux (Mandrake 9.0) 2.4.19-16mdk
Machine  : Intel IV (2.4 Ghz)
	 : 256 MB
	 

Subject : Replication

Files Attached: error.err, my.cnf

Description:
I am using replication feature of mysql.
I have 3 servers connected in a circular fashion. (A--->B---->C---->A)
Everything worked fine for a sometime (approximately 1 month) but now i have some problem.
The slave on machine B giving problems.
The machine B(master) for C(slave) running and working fine.
The machine C(master) for A(slave) running and working fine.
The machine A(master) for B(slave) NOT RUNNING.

To check the status of the master and the slave on each server I used to do the following.

SHOW MASTER STATUS
SHOW SLAVE STATUS

Then used to check whether the files are getting updated or not.
But few days back i found some problem.
On one of the machine the status was the same when i did SHOW SLAVE STATUS
The Master_Log_File showed "replication.029" and the Read_Master_Log_Pos showed "21252".
The Slave_IO_Running was "Yes" and Slave_Sql_Running was also "Yes".
This status remained continously with no error in the error file.

On running the command "SHOW PROCESSLIST" on the master server the BINLOG DUMP was NOT running.
where as in the log file it showed 
"030709 15:08:49  Slave I/O thread: connected to master 'username@machinename:3306'  replication started in log 'replication.029' at position 21252"

OBSERVATIONS:
On the master bin log file replication.029 was overwritten by replication.030,replication.031,replication.032,replication.033,replication.034
that means replication.029 got corrupted.
According to the manual when a slave starts it sends a signal to the master and then a process "BINLOG DUMP" starts running on the master server.
The slave shows "Slave_IO_Running=Yes" and the master not showing BINLOG DUMP command when doing "SHOW PROCESSLIST".
No error being showed on the slave machine(error file) also.

I again reset the slave(changed master_log_file and master_log_pos) and still no error coming but also the master "BINLOG DUMP" not working.
Please also let me know the file format for the binlog file.Is it encoded.

I dont know wht to do.Its urgent.
Please suggest

ERROR FILE

030624 14:38:52  mysqld started
Warning: Ignoring user change to 'root' because the user was set to 'root' earlier on the command line
/usr/sbin/mysqld: ready for connections.
Version: '4.0.12-log'  socket: '/var/lib/mysql/mysql.sock'  port: 3306
030624 14:39:01  InnoDB: Database was not shut down normally.
InnoDB: Starting recovery from log files...
InnoDB: Starting log scan based on checkpoint at
InnoDB: log sequence number 0 922085917
InnoDB: Doing recovery: scanned up to log sequence number 0 922085917
InnoDB: In a MySQL replication slave the last master binlog file
InnoDB: position 0 1410965, file name replication.017
InnoDB: Last MySQL binlog file position 0 1408688, file name /home/mysql/logs_repl/replication.014
030624 14:39:02  InnoDB: Flushing modified pages from the buffer pool...
030624 14:39:02  InnoDB: Started
030624 14:39:05  Slave I/O thread: error connecting to master 'rbcl@202.88.161.141:3306': Error: 'Can't connect to MySQL server on '202.88.161.141' (113)'  errno: 2003  retry-time: 900  retries: 86400
030624 15:39:56  Slave I/O thread: connected to master 'rbcl@202.88.161.141:3306',  replication started in log 'replication.017' at position 1410965
030624 16:40:09  Slave I/O thread: error reconnecting to master 'rbcl@202.88.161.141:3306': Error: 'Can't connect to MySQL server on '202.88.161.141' (113)'  errno: 2003  retry-time: 900  retries: 86400
030624 20:00:31  Slave I/O thread: error reconnecting to master 'rbcl@202.88.161.141:3306': Error: 'Lost connection to MySQL server during query'  errno: 2013  retry-time: 900  retries: 86400
030625 12:57:26  mysqld started
Warning: Ignoring user change to 'root' because the user was set to 'root' earlier on the command line
/usr/sbin/mysqld: ready for connections.
Version: '4.0.12-log'  socket: '/var/lib/mysql/mysql.sock'  port: 3306
030625 12:57:34  InnoDB: Database was not shut down normally.
InnoDB: Starting recovery from log files...
InnoDB: Starting log scan based on checkpoint at
InnoDB: log sequence number 0 926717367
InnoDB: Doing recovery: scanned up to log sequence number 0 926717367
InnoDB: 1 transaction(s) which must be rolled back or cleaned up
InnoDB: in total 1 row operations to undo
InnoDB: Trx id counter is 0 4632832
InnoDB: Starting rollback of uncommitted transactions
InnoDB: Rolling back trx with id 0 4632440, 1 rows to undo
InnoDB: Rolling back of trx id 0 4632440 completed
InnoDB: Rollback of uncommitted transactions completed
InnoDB: In a MySQL replication slave the last master binlog file
InnoDB: position 0 127497, file name replication.018
InnoDB: Last MySQL binlog file position 0 287016, file name /home/mysql/logs_repl/replication.015
030625 12:57:35  InnoDB: Flushing modified pages from the buffer pool...
030625 12:57:36  InnoDB: Started
030625 12:57:38  Slave I/O thread: error connecting to master 'rbcl@202.88.161.141:3306': Error: 'Can't connect to MySQL server on '202.88.161.141' (113)'  errno: 2003  retry-time: 900  retries: 86400
030625 14:57:59  Slave I/O thread: connected to master 'rbcl@202.88.161.141:3306',  replication started in log 'replication.018' at position 127566
030625 19:25:56  Slave: received 0 length packet from server, apparent master shutdown: 
030625 19:25:56  Slave I/O thread: Failed reading log event, reconnecting to retry, log 'replication.019' position 54938
030625 19:25:57  Slave I/O thread: error reconnecting to master 'rbcl@202.88.161.141:3306': Error: 'Can't connect to MySQL server on '202.88.161.141' (111)'  errno: 2003  retry-time: 900  retries: 86400
030626 13:29:31  mysqld started
Warning: Ignoring user change to 'root' because the user was set to 'root' earlier on the command line
/usr/sbin/mysqld: ready for connections.
Version: '4.0.12-log'  socket: '/var/lib/mysql/mysql.sock'  port: 3306
030626 13:29:39  InnoDB: Database was not shut down normally.
InnoDB: Starting recovery from log files...
InnoDB: Starting log scan based on checkpoint at
InnoDB: log sequence number 0 935142513
InnoDB: Doing recovery: scanned up to log sequence number 0 935142513
InnoDB: In a MySQL replication slave the last master binlog file
InnoDB: position 0 2662134, file name replication.020
InnoDB: Last MySQL binlog file position 0 374866, file name /home/mysql/logs_repl/replication.016
030626 13:29:40  InnoDB: Flushing modified pages from the buffer pool...
030626 13:29:41  InnoDB: Started
030626 13:29:43  Slave I/O thread: error connecting to master 'rbcl@202.88.161.141:3306': Error: 'Can't connect to MySQL server on '202.88.161.141' (113)'  errno: 2003  retry-time: 900  retries: 86400
030626 14:31:22  mysqld started
Warning: Ignoring user change to 'root' because the user was set to 'root' earlier on the command line
030626 14:31:32  InnoDB: Database was not shut down normally.
InnoDB: Starting recovery from log files...
InnoDB: Starting log scan based on checkpoint at
InnoDB: log sequence number 0 935142513
InnoDB: Doing recovery: scanned up to log sequence number 0 935142513
InnoDB: In a MySQL replication slave the last master binlog file
InnoDB: position 0 2662134, file name replicati/usr/sbin/mysqld: ready for connections.
Version: '4.0.12-log'  socket: '/var/lib/mysql/mysql.sock'  port: 3306
on.020
InnoDB: Last MySQL binlog file position 0 374866, file name /home/mysql/logs_repl/replication.016
030626 14:31:33  InnoDB: Flushing modified pages from the buffer pool...
030626 14:31:33  InnoDB: Started
030626 14:31:36  Slave I/O thread: connected to master 'rbcl@202.88.161.141:3306',  replication started in log 'replication.020' at position 2662197
030627 12:38:57  mysqld started
Warning: Ignoring user change to 'root' because the user was set to 'root' earlier on the command line
/usr/sbin/mysqld: ready for connections.
Version: '4.0.12-log'  socket: '/var/lib/mysql/mysql.sock'  port: 3306
030627 12:39:06  InnoDB: Database was not shut down normally.
InnoDB: Starting recovery from log files...
InnoDB: Starting log scan based on checkpoint at
InnoDB: log sequence number 0 935476042
InnoDB: Doing recovery: scanned up to log sequence number 0 935476042
InnoDB: In a MySQL replication slave the last master binlog file
InnoDB: position 0 231180, file name replication.021
InnoDB: Last MySQL binlog file position 0 103311, file name /home/mysql/logs_repl/replication.018
030627 12:39:07  InnoDB: Flushing modified pages from the buffer pool...
030627 12:39:07  InnoDB: Started
030627 12:39:09  Slave I/O thread: error connecting to master 'rbcl@202.88.161.141:3306': Error: 'Can't connect to MySQL server on '202.88.161.141' (113)'  errno: 2003  retry-time: 900  retries: 86400
030627 14:54:31  Slave I/O thread: connected to master 'rbcl@202.88.161.141:3306',  replication started in log 'replication.021' at position 103242
030627 23:39:57  Slave I/O thread: error reconnecting to master 'rbcl@202.88.161.141:3306': Error: 'Can't connect to MySQL server on '202.88.161.141' (115)'  errno: 2003  retry-time: 900  retries: 86400
030628 13:47:53  mysqld started
Warning: Ignoring user change to 'root' because the user was set to 'root' earlier on the command line
/usr/sbin/mysqld: ready for connections.
Version: '4.0.12-log'  socket: '/var/lib/mysql/mysql.sock'  port: 3306
030628 13:48:02  InnoDB: Database was not shut down normally.
InnoDB: Starting recovery from log files...
InnoDB: Starting log scan based on checkpoint at
InnoDB: log sequence number 0 935493819
InnoDB: Doing recovery: scanned up to log sequence number 0 935493819
InnoDB: In a MySQL replication slave the last master binlog file
InnoDB: position 0 2168113, file name replication.022
InnoDB: Last MySQL binlog file position 0 17618, file name /home/mysql/logs_repl/replication.019
030628 13:48:03  InnoDB: Flushing modified pages from the buffer pool...
030628 13:48:03  InnoDB: Started
030628 13:48:05  Slave I/O thread: error connecting to master 'rbcl@202.88.161.141:3306': Error: 'Can't connect to MySQL server on '202.88.161.141' (113)'  errno: 2003  retry-time: 900  retries: 86400
030628 17:07:49  Slave I/O thread: connected to master 'rbcl@202.88.161.141:3306',  replication started in log 'replication.022' at position 2168113
030630 13:14:15  mysqld started
Warning: Ignoring user change to 'root' because the user was set to 'root' earlier on the command line
/usr/sbin/mysqld: ready for connections.
Version: '4.0.12-log'  socket: '/var/lib/mysql/mysql.sock'  port: 3306
030630 13:14:24  InnoDB: Database was not shut down normally.
InnoDB: Starting recovery from log files...
InnoDB: Starting log scan based on checkpoint at
InnoDB: log sequence number 0 935782736
InnoDB: Doing recovery: scanned up to log sequence number 0 935782736
InnoDB: In a MySQL replication slave the last master binlog file
InnoDB: position 0 330059, file name replication.023
InnoDB: Last MySQL binlog file position 0 177794, file name /home/mysql/logs_repl/replication.020
030630 13:14:25  InnoDB: Flushing modified pages from the buffer pool...
030630 13:14:25  InnoDB: Started
030630 13:14:28  Slave I/O thread: error connecting to master 'rbcl@202.88.161.141:3306': Error: 'Can't connect to MySQL server on '202.88.161.141' (113)'  errno: 2003  retry-time: 900  retries: 86400
030630 14:44:43  Slave I/O thread: connected to master 'rbcl@202.88.161.141:3306',  replication started in log 'replication.023' at position 330059
030701 13:07:42  mysqld started
Warning: Ignoring user change to 'root' because the user was set to 'root' earlier on the command line
/usr/sbin/mysqld: ready for connections.
Version: '4.0.12-log'  socket: '/var/lib/mysql/mysql.sock'  port: 3306
030701 13:07:50  InnoDB: Database was not shut down normally.
InnoDB: Starting recovery from log files...
InnoDB: Starting log scan based on checkpoint at
InnoDB: log sequence number 0 936044011
InnoDB: Doing recovery: scanned up to log sequence number 0 936044011
InnoDB: In a MySQL replication slave the last master binlog file
InnoDB: position 0 107145, file name replication.024
InnoDB: Last MySQL binlog file position 0 106041, file name /home/mysql/logs_repl/replication.021
030701 13:07:51  InnoDB: Flushing modified pages from the buffer pool...
030701 13:07:51  InnoDB: Started
030701 13:07:54  Slave I/O thread: error connecting to master 'rbcl@202.88.161.141:3306': Error: 'Can't connect to MySQL server on '202.88.161.141' (113)'  errno: 2003  retry-time: 900  retries: 86400
030701 15:23:29  Slave I/O thread: connected to master 'rbcl@202.88.161.141:3306',  replication started in log 'replication.024' at position 107973
030701 16:50:00  Slave I/O thread: error reconnecting to master 'rbcl@202.88.161.141:3306': Error: 'Can't connect to MySQL server on '202.88.161.141' (113)'  errno: 2003  retry-time: 900  retries: 86400
030701 20:06:58  Slave I/O thread: error reconnecting to master 'rbcl@202.88.161.141:3306': Error: 'Can't connect to MySQL server on '202.88.161.141' (113)'  errno: 2003  retry-time: 900  retries: 86400
030702 12:54:38  mysqld started
Warning: Ignoring user change to 'root' because the user was set to 'root' earlier on the command line
/usr/sbin/mysqld: ready for connections.
Version: '4.0.12-log'  socket: '/var/lib/mysql/mysql.sock'  port: 3306
030702 12:54:46  InnoDB: Database was not shut down normally.
InnoDB: Starting recovery from log files...
InnoDB: Starting log scan based on checkpoint at
InnoDB: log sequence number 0 936660097
InnoDB: Doing recovery: scanned up to log sequence number 0 936660097
InnoDB: In a MySQL replication slave the last master binlog file
InnoDB: position 0 314853, file name replication.026
InnoDB: Last MySQL binlog file position 0 312967, file name /home/mysql/logs_repl/replication.022
030702 12:54:47  InnoDB: Flushing modified pages from the buffer pool...
030702 12:54:47  InnoDB: Started
030702 12:54:50  Slave I/O thread: error connecting to master 'rbcl@202.88.161.141:3306': Error: 'Can't connect to MySQL server on '202.88.161.141' (113)'  errno: 2003  retry-time: 900  retries: 86400
030702 15:10:06  Slave I/O thread: connected to master 'rbcl@202.88.161.141:3306',  replication started in log 'replication.026' at position 314853
030703 12:56:53  mysqld started
Warning: Ignoring user change to 'root' because the user was set to 'root' earlier on the command line
/usr/sbin/mysqld: ready for connections.
Version: '4.0.12-log'  socket: '/var/lib/mysql/mysql.sock'  port: 3306
030703 12:57:02  InnoDB: Database was not shut down normally.
InnoDB: Starting recovery from log files...
InnoDB: Starting log scan based on checkpoint at
InnoDB: log sequence number 0 936984091
InnoDB: Doing recovery: scanned up to log sequence number 0 936984091
InnoDB: In a MySQL replication slave the last master binlog file
InnoDB: position 0 257570, file name replication.027
InnoDB: Last MySQL binlog file position 0 250739, file name /home/mysql/logs_repl/replication.023
030703 12:57:03  InnoDB: Flushing modified pages from the buffer pool...
030703 12:57:03  InnoDB: Started
030703 12:57:05  Slave I/O thread: error connecting to master 'rbcl@202.88.161.141:3306': Error: 'Can't connect to MySQL server on '202.88.161.141' (113)'  errno: 2003  retry-time: 900  retries: 86400
030703 14:42:25  Slave I/O thread: connected to master 'rbcl@202.88.161.141:3306',  replication started in log 'replication.027' at position 257570
030704 14:00:13  mysqld started
Warning: Ignoring user change to 'root' because the user was set to 'root' earlier on the command line
/usr/sbin/mysqld: ready for connections.
Version: '4.0.12-log'  socket: '/var/lib/mysql/mysql.sock'  port: 3306
030704 14:00:21  InnoDB: Database was not shut down normally.
InnoDB: Starting recovery from log files...
InnoDB: Starting log scan based on checkpoint at
InnoDB: log sequence number 0 937033542
InnoDB: Doing recovery: scanned up to log sequence number 0 937033542
InnoDB: In a MySQL replication slave the last master binlog file
InnoDB: position 0 22058, file name replication.028
InnoDB: Last MySQL binlog file position 0 21422, file name /home/mysql/logs_repl/replication.024
030704 14:00:22  InnoDB: Flushing modified pages from the buffer pool...
030704 14:00:23  InnoDB: Started
030704 14:00:26  Slave I/O thread: error connecting to master 'rbcl@202.88.161.141:3306': Error: 'Can't connect to MySQL server on '202.88.161.141' (113)'  errno: 2003  retry-time: 900  retries: 86400
030704 15:00:34  Slave I/O thread: connected to master 'rbcl@202.88.161.141:3306',  replication started in log 'replication.028' at position 22127
030705 12:38:47  mysqld started
Warning: Ignoring user change to 'root' because the user was set to 'root' earlier on the command line
/usr/sbin/mysqld: ready for connections.
Version: '4.0.12-log'  socket: '/var/lib/mysql/mysql.sock'  port: 3306
030705 12:38:56  InnoDB: Database was not shut down normally.
InnoDB: Starting recovery from log files...
InnoDB: Starting log scan based on checkpoint at
InnoDB: log sequence number 0 937462308
InnoDB: Doing recovery: scanned up to log sequence number 0 937462308
InnoDB: 1 transaction(s) which must be rolled back or cleaned up
InnoDB: in total 8 row operations to undo
InnoDB: Trx id counter is 0 4659968
InnoDB: Starting rollback of uncommitted transactions
InnoDB: Rolling back trx with id 0 4659551, 8 rows to undo
InnoDB: Rolling back of trx id 0 4659551 completed
InnoDB: Rollback of uncommitted transactions completed
InnoDB: In a MySQL replication slave the last master binlog file
InnoDB: position 0 21114, file name replication.029
InnoDB: Last MySQL binlog file position 0 295367, file name /home/mysql/logs_repl/replication.025
030705 12:38:58  InnoDB: Flushing modified pages from the buffer pool...
030705 12:38:58  InnoDB: Started
030705 12:39:02  Slave I/O thread: error connecting to master 'rbcl@202.88.161.141:3306': Error: 'Can't connect to MySQL server on '202.88.161.141' (113)'  errno: 2003  retry-time: 900  retries: 86400
030705 16:54:02  mysqld started
Warning: Ignoring user change to 'root' because the user was set to 'root' earlier on the command line
/usr/sbin/mysqld: ready for connections.
Version: '4.0.12-log'  socket: '/var/lib/mysql/mysql.sock'  port: 3306
030705 16:54:11  InnoDB: Database was not shut down normally.
InnoDB: Starting recovery from log files...
InnoDB: Starting log scan based on checkpoint at
InnoDB: log sequence number 0 937470405
InnoDB: Doing recovery: scanned up to log sequence number 0 937470405
InnoDB: In a MySQL replication slave the last master binlog file
InnoDB: position 0 21114, file name replication.029
InnoDB: Last MySQL binlog file position 0 4862, file name /home/mysql/logs_repl/replication.026
030705 16:54:12  InnoDB: Flushing modified pages from the buffer pool...
030705 16:54:12  InnoDB: Started
030705 16:54:14  Slave I/O thread: error connecting to master 'rbcl@202.88.161.141:3306': Error: 'Can't connect to MySQL server on '202.88.161.141' (113)'  errno: 2003  retry-time: 900  retries: 86400
030705 18:55:18  Slave I/O thread: connected to master 'rbcl@202.88.161.141:3306',  replication started in log 'replication.029' at position 21252
030705 23:41:01  Slave I/O thread: error reconnecting to master 'rbcl@202.88.161.141:3306': Error: 'Can't connect to MySQL server on '202.88.161.141' (115)'  errno: 2003  retry-time: 900  retries: 86400
030707 12:34:21  mysqld started
Warning: Ignoring user change to 'root' because the user was set to 'root' earlier on the command line
030707 12:34:30/usr/sbin/mysqld: ready for connections.
Version: '4.0.12-log'  socket: '/var/lib/mysql/mysql.sock'  port: 3306
  InnoDB: Database was not shut down normally.
InnoDB: Starting recovery from log files...
InnoDB: Starting log scan based on checkpoint at
InnoDB: log sequence number 0 937470405
InnoDB: Doing recovery: scanned up to log sequence number 0 937470405
InnoDB: In a MySQL replication slave the last master binlog file
InnoDB: position 0 21114, file name replication.029
InnoDB: Last MySQL binlog file position 0 4862, file name /home/mysql/logs_repl/replication.026
030707 12:34:31  InnoDB: Flushing modified pages from the buffer pool...
030707 12:34:31  InnoDB: Started
030707 12:34:33  Slave I/O thread: error connecting to master 'rbcl@202.88.161.141:3306': Error: 'Can't connect to MySQL server on '202.88.161.141' (113)'  errno: 2003  retry-time: 900  retries: 86400
030707 14:41:00  mysqld started
Warning: Ignoring user change to 'root' because the user was set to 'root' earlier on the command line
/usr/sbin/mysqld: ready for connections.
Version: '4.0.12-log'  socket: '/var/lib/mysql/mysql.sock'  port: 3306
030707 14:41:09  InnoDB: Database was not shut down normally.
InnoDB: Starting recovery from log files...
InnoDB: Starting log scan based on checkpoint at
InnoDB: log sequence number 0 937470405
InnoDB: Doing recovery: scanned up to log sequence number 0 937470405
InnoDB: In a MySQL replication slave the last master binlog file
InnoDB: position 0 21114, file name replication.029
InnoDB: Last MySQL binlog file position 0 4862, file name /home/mysql/logs_repl/replication.026
030707 14:41:10  InnoDB: Flushing modified pages from the buffer pool...
030707 14:41:10  InnoDB: Started
030707 14:41:12  Slave I/O thread: error connecting to master 'rbcl@202.88.161.141:3306': Error: 'Can't connect to MySQL server on '202.88.161.141' (113)'  errno: 2003  retry-time: 900  retries: 86400
030707 17:11:29  Slave I/O thread: connected to master 'rbcl@202.88.161.141:3306',  replication started in log 'replication.029' at position 21252
030708  1:43:20  Slave I/O thread: error reconnecting to master 'rbcl@202.88.161.141:3306': Error: 'Can't connect to MySQL server on '202.88.161.141' (115)'  errno: 2003  retry-time: 900  retries: 86400
030708 17:58:44  /usr/sbin/mysqld: Normal shutdown

030708 17:58:44  Slave I/O thread exiting, read up to log 'replication.029', position 21252
030708 17:58:45  InnoDB: Starting shutdown...
030708 17:58:48  InnoDB: Shutdown completed
030708 17:58:48  /usr/sbin/mysqld: Shutdown Complete

030708 17:58:48  mysqld ended

030708 17:59:02  mysqld started
Warning: Ignoring user change to 'root' because the user was set to 'root' earlier on the command line
/usr/sbin/mysqld: ready for connections.
Version: '4.0.12-log'  socket: '/var/lib/mysql/mysql.sock'  port: 3306
030708 17:59:05  InnoDB: Started
030708 17:59:08  Slave I/O thread: connected to master 'rbcl@202.88.161.141:3306',  replication started in log 'replication.029' at position 21252
030708 20:38:23  Slave I/O thread exiting, read up to log 'replication.029', position 21252
030708 20:39:05  Slave I/O thread: connected to master 'rbcl@202.88.161.141:3306',  replication started in log 'replication.029' at position 21252
030709  1:24:50  Slave I/O thread: error reconnecting to master 'rbcl@202.88.161.141:3306': Error: 'Can't connect to MySQL server on '202.88.161.141' (115)'  errno: 2003  retry-time: 900  retries: 86400
030709 13:53:24  mysqld started
Warning: Ignoring user change to 'root' because the user was set to 'root' earlier on the command line
030709 13:53:33  InnoDB: Database was not shut down normally.
InnoDB: Starting recovery from log files...
InnoDB: Starting log scan based on checkpoint at
InnoDB: log sequence number 0 937523003
InnoDB: Doing recovery: scanned up to log sequence number 0 937523003
InnoDB: In a MySQL replication slave the last master binlog file
InnoDB: position 0 21114, file name replication.029
InnoDB: Last MySQL binlog file position 0 3201, file name /home/mysql/logs_repl/replication.030
030709 13:53:34  InnoDB: Flushi/usr/sbin/mysqld: ready for connections.
Version: '4.0.12-log'  socket: '/var/lib/mysql/mysql.sock'  port: 3306
ng modified pages from the buffer pool...
030709 13:53:34  InnoDB: Started
030709 13:53:37  Slave I/O thread: error connecting to master 'rbcl@202.88.161.141:3306': Error: 'Can't connect to MySQL server on '202.88.161.141' (113)'  errno: 2003  retry-time: 900  retries: 86400
030709 15:08:49  Slave I/O thread: connected to master 'rbcl@202.88.161.141:3306',  replication started in log 'replication.029' at position 21252

MY.CNF

[mysqld]
# You can write your other MySQL server options here

user=root
set-variable = max_connections=300
transaction-isolation=READ-COMMITTED
set-variable = connect_timeout=30
set-variable = wait_timeout=3600
set-variable = interactive_timeout=3600

default-table-type=innodb

server-id = 30
#(Server-id for three servers is different)
binlog-do-db=rbcl
log-slave-updates

log-bin=/home/mysql/logs_repl/replication.log
log-error=/home/mysql/logs_gen/error.err
log=/home/mysql/logs_gen/genlog.log
log-update=/home/mysql/logs_upd/update.log
relay-log=/home/mysql/logs_relay/relay.log

replicate-wild-ignore-table=rbcl.%temp%
replicate-wild-ignore-table=rbcl.transaction_no
replicate-wild-ignore-table=rbcl.%tmp%

master-connect-retry=900

How to repeat:
No Idea

Suggested fix:
NO IDEA

First, an unrelated note: the error log shows an important number of unclean shutdowns of MySQL. I hope this does not occur too frequently in the future, or you may one day experience replication corruption (i.e. the slaves not having time to flush master.info and relay-log.info before stopping, which will make the slaves, at restart, replicate a query they have already replicated), which can explain many bizarre (looking like bugs) scenarios.
About your bug:

030709 13:53:37  Slave I/O thread: error connecting to master
'rbcl@202.88.161.141:3306': Error: 'Can't connect to MySQL server on
'202.88.161.141' (113)'  errno: 2003  retry-time: 900  retries: 86400
030709 15:08:49  Slave I/O thread: connected to master
'rbcl@202.88.161.141:3306',  replication started in log
'replication.029' at position 21252

This says the slave was able to finally connect to the master.

"On the master bin log file replication.029 was overwritten by
replication.030,replication.031,replication.032,replication.033,replication.034
that means replication.029 got corrupted."

Sorry, what do you exactly mean here? Do you mean replication.029 did not exist on the master anymore ? Then there was .030, and later .030 did not exist anymore and there was .031 instead ?

The replication.029 existed. The machine on which it was (master) had some changes in the database which were recorded there.
But later when i came to knw that the replication not working i went and checked the replication.029. i did the following 
SHOW BINLOG EVENTS in 'replication.029'.
This gave me the following message.
Error when executing command SHOW BINLOG EVENTS.Binlog has bad magic number;
It's not a binry log file that can be used by this version of MySQL.

Then i went and checked the file.
The contents of replication.030 were there in this file.
when i reached the end of the file of replication.030 in replciation.029,
then contents of replication.031 started and this continued till replication.034.

The files replication.030 to replication.034 still exists and are not corrupted,
i can read them by the "SHOW BINLOG EVENTS" command.

What i can make out is whenever the logs were written, they were written to 2 files i.e replication.029 and other respective files.
Though there is no data loss from replication.030,replication.031,replication.032,replication.034
If u want i can mail you the files replication.029 to replication.034.
just provide me your mail id.

Yes, if you can, please make an archive of your binlogs (including the complete replication.index file, and the complete mysqld's error log), give it a name that contains the "bug823" string and ftp it there:
 ftp://support.mysql.com/pub/mysql/secret/
Thank you!
Btw, binlogs can be read with mysqlbinlog.

Later i reset the slave master_log_file and master_log_pos and it started replicating.
But my problem is why did it overwrite the file.
And pls if u could please tell me the format of the binlog files.
Thank You

Thank you for taking the time to report a problem. Unfortunately
you are not using a current version of the product your reported a
problem with -- the problem might already be fixed. Please download
a new version from http://www.mysql.com/downloads/

If you are able to reproduce the bug with one of the latest versions,
please change the version on this bug report to the version you
tested and change the status back to "Open". Again, thank you for
your continued support of MySQL.

>ater i reset the slave master_log_file and master_log_pos and it started
>replicating.
>But my problem is why did it overwrite the file.

I'm sorry, but your first very serious problem is all these abnormal shutdowns you have. This is what you must first investigate. It can explain whatever problem you get.
From the error logs, it looks like most MySQL shutdowns on your machine are abnormal: a normal shutdown has a "mysqld ended" line.

030612 20:33:04 Slave I/O thread: error reconnecting to master 'rbcl@202.88.161.141:3306': Error: 'Can't conn
ect to MySQL server on '202.88.161.141' (113)' errno: 2003 retry-time: 900 retries: 86400

030613 13:00:43 mysqld started

See? it started, but shows no message about a prior shutdown!
And InnoDB says it too:
030613 13:00:50 InnoDB: Database was not shut down normally.

Even the binlogs say it, because (when you do mysqlbinlog replication.010) the last line does not look like this:
030710 22:12:50 server id 1 log_pos 1769 Stop

In the error log you also have some
030710 12:05:42 Error in Log_event::read_log_event(): 'Event too small', data_len=0,event_type=0
which show that either the binlog is corrupted, or, in case of a slave, the master.info or relay-log.info is corrupted. Which can all be explained by a brutal mysqld shutdown.

Even the corruption of replication.029 can be explained by a bad shutdown.

You have to first get rid of all these unclean shutdowns. Reasons for these unclean shutdowns can be:
- you issue 'kill -9' on mysqld (I doubt it).
- there's a brutal electrical power shutdown of the machine.
- if mysqld is started and stopped by /etc/init.d/mysql, maybe this script is not called in the right runlevels. Bad runlevels of /etc/init.d/mysql in our RPM binary, installed in Redhat & Mandrake, have been reported:
http://bugs.mysql.com/bug.php?id=272
This can be fixed by upgrading to our 4.0.14 RPM binary, which will be released in a few weeks. I'm almost sure this could be the problem: if /etc/init.d/mysql is not registered for the right runlevels, then at machine halt, the inittab process may issue 'kill -9' on mysqld.
You can check if runlevels are good with
chkconfig --list mysql (login as root).
A buggy result of this command could be like this:
[root@gbichot2 logiciels]# chkconfig --list mysql
mysql 0:off 1:off 2:off 3:off 4:off 5:off 6:off

A good result should look like:
mysql 0:off 1:off 2:on 3:on 4:on 5:on 6:off

Look at http://bugs.mysql.com/bug.php?id=272 for the way to solve this until 4.0.14 is released.

>And pls if u could please tell me the format of the binlog files.

It's a binary format, you can extract readable information using the 'mysqlbinlog' program which is included in MySQL (if you're using RPM, it's in the 'client' RPM).