Bug #17256 Intermittent errors stop slave from processing with NDB is set as default
Submitted: 9 Feb 2006 1:21 Modified: 13 Jul 2006 17:40
Reporter: Jonathan Miller Email Updates:
Status: Closed Impact on me:
None 
Category:MySQL Server: Replication Severity:S2 (Serious)
Version:5.1.7 OS:Linux (Linux 32 Bit OS)
Assigned to: Mats Kindahl CPU Architecture:Any

[9 Feb 2006 1:21] Jonathan Miller
Description:
Sorry buys I tried to narrow this one down for you, but I only get it all the time if the whole test is enabled. 

TEST                            RESULT
-------------------------------------------------------
rpl_multi_update3              [ fail ]

Errors are (from /home/ndbdev/jmiller/clones/mysql-5.1-new/mysql-test/var/log/mysqltest-time) :
mysqltest: In included file "./extra/rpl_tests/rpl_multi_update3.test": At line 154: could not sync with master ('select master_pos_wait('master-bin.000001', 3531)' returned NULL)
(the last lines may be the most important ones)

Slave error log:
060209  4:14:05 [ERROR] Slave: Error in Update_rows event: row application failed, Error_code: 137
060209  4:14:05 [ERROR] Slave: Error in Update_rows event: error during transaction execution on table test.t1, Error_code: 137
060209  4:14:05 [ERROR] Slave (additional info): Unknown error Error_code: 1105
060209  4:14:05 [Warning] Slave: Unknown error Error_code: 1105
060209  4:14:05 [Warning] Slave: Unknown error Error_code: 1105
060209  4:14:05 [ERROR] Error running query, slave SQL thread aborted. Fix the problem, and restart the slave SQL thread with "SLAVE START". We stopped at log 'master-bin.000001' position 1272

../extra/perror 137
MySQL error code 137: No more records (read after end of file)

How to repeat:
./mysql-test-run --force --ndb-extra-test --with-ndbcluster-all --mysqld=--default-storage-engine=ndb --do-test=rpl_multi_update3
[9 Feb 2006 14:24] Jonathan Miller
I feel that this failure is in the same area eventhough it is delete and not update.
TEST                            RESULT
-------------------------------------------------------
rpl_sp_effects                 [ fail ]

Errors are (from /home/ndbdev/jmiller/clones/mysql-5.1-new/mysql-test/var/log/mysqltest-time) :
mysqltest: At line 48: could not sync with master ('select master_pos_wait('master-bin.000001', 2147)' returned NULL)
(the last lines may be the most important ones)

060209 17:20:38 [ERROR] Slave: Error in Delete_rows event: row application failed, Error_code: 137
060209 17:20:38 [ERROR] Slave: Error in Delete_rows event: error during transaction execution on table test.t1, Error_code: 137
060209 17:20:38 [ERROR] Slave (additional info): Unknown error Error_code: 1105
060209 17:20:38 [Warning] Slave: Unknown error Error_code: 1105
060209 17:20:38 [Warning] Slave: Unknown error Error_code: 1105
060209 17:20:38 [ERROR] Error running query, slave SQL thread aborted. Fix the problem, and restart the slave SQL thread with "SLAVE START". We stopped at log 'master-bin.000001' position 1285

./mysql-test-run --force --ndb-extra-test --with-ndbcluster-all --mysqld=--default-storage-engine=ndb --do-test=rpl_sp_effects
[24 Feb 2006 13:11] Joerg Bruehe
Telling from the symptoms, these might be the same thing ?
But we got them even when the test suite was started "--skip-ndbcluster",
so in the runs 
rhas3-x86-glibc23-5.1.log   normal
rhas3-x86-glibc23-5.1.log   ps

The "rpl_row_*" tests were skipped in the "normal" and "ps" runs, so they failed in
rhas3-x86-glibc23-5.1.log   ps+rowrepl+NDB

All these tests passed in the "rowrepl" (no "ps", no NDB) case.

Detected by running the test suite on 5.1.7-beta, build based on ChangeSet
  1.2156.2.1 06/02/24 00:29:50 andrey@lmy004. +3 -0
  fix for bug #16400

All specific to machine "rhas3-x86".

=====
rpl_insert_id                  [ fail ]

Errors are (from PATH/mysqltest-time) :
mysqltest: In included file "./extra/rpl_tests/rpl_insert_id.test": At line 47: could not sync with master ('select master_
pos_wait('master-bin.000001', 1724)' returned NULL)
(the last lines may be the most important ones)
=====

=====
rpl_insert_ignore              [ fail ]

Errors are (from PATH/mysqltest-time) :
mysqltest: In included file "./extra/rpl_tests/rpl_insert_ignore.test": At line 42: could not sync with master ('select mas
ter_pos_wait('master-bin.000001', 1964)' returned NULL)
(the last lines may be the most important ones)
=====

=====
rpl_relayrotate                [ fail ]

Errors are (from PATH/mysqltest-time) :
mysqltest: In included file "./extra/rpl_tests/rpl_relayrotate.test": At line 56: could not sync with master ('select maste
r_pos_wait('master-bin.000001', 735190)' returned NULL)
(the last lines may be the most important ones)
=====

=====
rpl_relay_space_innodb         [ fail ]

Errors are (from PATH/mysqltest-time) :
mysqltest: In included file "./extra/rpl_tests/rpl_sv_relay_space.test": At line 25: could not sync with master ('select ma
ster_pos_wait('master-bin.000001', 764)' returned NULL)
(the last lines may be the most important ones)
=====

=====
rpl_row_basic_3innodb          [ fail ]

Errors are (from PATH/mysqltest-time) :
mysqltest: In included file "./include/rpl_row_basic.inc": At line 15: could not sync with master ('select master_pos_wait(
'master-bin.000001', 235)' returned NULL)
(the last lines may be the most important ones)
=====

=====
rpl_row_blob_innodb            [ fail ]

Errors are (from PATH/mysqltest-time) :
mysqltest: In included file "./extra/rpl_tests/rpl_row_blob.test": At line 42: could not sync with master ('select master_p
os_wait('master-bin.000001', 18169)' returned NULL)
(the last lines may be the most important ones)
=====

=====
rpl_row_log_innodb             [ fail ]

Errors are (from PATH/mysqltest-time) :
mysqltest: In included file "./extra/rpl_tests/rpl_log.test": At line 80: could not sync with master ('select master_pos_wa
it('master-bin.000002', 201)' returned NULL)
(the last lines may be the most important ones)
=====

=====
rpl_row_sp002_innodb           [ fail ]

Errors are (from PATH/mysqltest-time) :
mysqltest: In included file "./extra/rpl_tests/rpl_row_sp002.test": At line 76: could not sync with master ('select master_
pos_wait('master-bin.000001', 2396)' returned NULL)
(the last lines may be the most important ones)
=====

=====
rpl_row_sp007_innodb           [ fail ]

Errors are (from PATH/mysqltest-time) :
mysqltest: In included file "./extra/rpl_tests/rpl_row_sp007.test": At line 47: could not sync with master ('select master_
pos_wait('master-bin.000001', 1040)' returned NULL)
(the last lines may be the most important ones)
=====
[13 Jul 2006 17:40] Jonathan Miller
Well, it seems to somewhat pass at this point. (order bys would help)
=======================================================
Starting Tests in the 'main' suite

TEST                           RESULT         TIME (ms)
-------------------------------------------------------

rpl_multi_update3              [ fail ]

Errors are (from /data1/mysql-5.1/mysql-test/var/log/mysqltest-
mysqltest: Result content mismatch
(the last lines may be the most important ones)
Below are the diffs between actual and expected results:
-------------------------------------------------------
*** r/rpl_multi_update3.result  2006-07-06 17:26:14.000000000 +
--- r/rpl_multi_update3.reject  2006-07-13 20:38:45.000000000 +
***************
*** 167,177 ****
  -- MASTER AFTER JOIN --
  select * from t1;
  idp   idpro   price
  1     1       3.0000
  2     2       1.0000
- 3     1       1.0000
  4     1       4.0000
! 5     3       2.0000
  6     2       4.0000
  select * from t2;
  idpro price   nbprice
--- 167,177 ----
  -- MASTER AFTER JOIN --
  select * from t1;
  idp   idpro   price
+ 5     3       2.0000
  1     1       3.0000
  2     2       1.0000
  4     1       4.0000
! 3     1       1.0000
  6     2       4.0000
  select * from t2;
  idpro price   nbprice
***************
*** 183,196 ****
  -- SLAVE AFTER JOIN --
  select * from t1;
  idp   idpro   price
  1     1       3.0000
  2     2       1.0000
- 3     1       1.0000
  4     1       4.0000
- 5     3       2.0000
- 6     2       4.0000
  select * from t2;
  idpro price   nbprice
  1     1.0000  3
  2     1.0000  2
- 3     2.0000  1
--- 183,196 ----
  -- SLAVE AFTER JOIN --
  select * from t1;
  idp   idpro   price
+ 5     3       2.0000
+ 3     1       1.0000
  1     1       3.0000
+ 6     2       4.0000
  2     2       1.0000
  4     1       4.0000
  select * from t2;
  idpro price   nbprice
+ 3     2.0000  1
  1     1.0000  3
  2     1.0000  2
-------------------------------------------------------