MySQL Bugs: #29904: Cluster replication breaks on large VARCHAR fields

Bug #29904	Cluster replication breaks on large VARCHAR fields
Submitted:	19 Jul 2007 15:43	Modified:	7 Jan 2008 14:38
Reporter:	Geert Vanderkelen	Email Updates:
Status:	Closed	Impact on me:	None
Category:	MySQL Cluster: Replication	Severity:	S2 (Serious)
Version:	5.1.18_6.2.2	OS:	Linux
Assigned to:	justin he	CPU Architecture:	Any

Description:
The slave will give an error after some records have been replicated. For example, if we insert 100000 records on the Master (which works), the Slave will fail at, for example, 2000.

The weird thing is that it seems to be depend on the size of the VARCHAR.

From MySQL Error log:

070719 17:28:38 [ERROR] Slave: Error in Write_rows event: row application failed, Error_code: 0
070719 17:28:38 [ERROR] Slave: Error in Write_rows event: error during transaction execution on table t
est.t1, Error_code: 1105
070719 17:28:38 [Warning] Slave: Got error 827 'Out of memory in Ndb Kernel, table data (increase DataM
emory)' from NDB Error_code: 1296
070719 17:28:38 [Warning] Slave: Unknown error Error_code: 1105
070719 17:28:38 [Warning] Slave: Unknown error Error_code: 1105
070719 17:28:38 [ERROR] Error running query, slave SQL thread aborted. Fix the problem, and restart the
 slave SQL thread with "SLAVE START". We stopped at log 'master-log.000007' position 329240997

Note that there is still enough Data and IndexMemory

How to repeat:
Setup Master and Slave cluster (configuration attached) and create a table like:

DROP TABLE IF EXISTS t1;
CREATE TABLE t1 (id INT UNSIGNED NOT NULL AUTO_INCREMENT, data VARCHAR(7500), PRIMARY KEY (id)) ENGINE=ndbcluster DEFAULT CHARSET=latin1;

And insert 100000 records in the table. I have done it with 5 threads doing each an INSERT..

I have made the VARCHAR smaller and found it working OK under 6270.. Around there I can't reproduce it all the time.

Configuration of Master and Slave (my.cnf's and config.ini)

Attachment: bug29904_config.txt (text/plain), 700 bytes.

Reproduced with 5.1.18_6.2.2 and 5.1.19_6.2.3 (MCCGE).
Could not reproduce with 5.1.20-beta.

Just to confirm: works fine with MyISAM.

Verified with 5.1.19_6.2 from BK: still not working, but different error now with table 't1' is full.

070829 11:20:33 [ERROR] Slave SQL: Error in Write_rows event: row application failed. The table 't1' is full, Error_code: 1114
070829 11:20:33 [ERROR] Slave SQL: Error in Write_rows event: error during transaction execution on table test.t1. The table 't1' is 
full, Error_code: 1114
070829 11:20:33 [Warning] Slave: Got error 827 'Out of memory in Ndb Kernel, table data (increase DataMemory)' from NDB Error_code: 1
296
070829 11:20:33 [Warning] Slave: The table 't1' is full Error_code: 1114
070829 11:20:33 [ERROR] Error running query, slave SQL thread aborted. Fix the problem, and restart the slave SQL thread with "SLAVE 
START". We stopped at log 'master-log.000017' position 1384507

can you try the following patch
===== storage/ndb/src/kernel/blocks/dbtup/Undo_buffer.cpp 1.5 vs edited =====
57c57
<     page= (UndoPage*)m_mm->alloc_page(RG_DISK_OPERATIONS, &m_first_free);
---
>     page= (UndoPage*)m_mm->alloc_page(RG_DATAMEM, &m_first_free);
109c109
<       m_mm->release_page(RG_DISK_OPERATIONS, key->m_page_no);
---
>       m_mm->release_page(RG_DATAMEM, key->m_page_no);

A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/33667

ChangeSet@1.2581, 2007-09-04 22:23:48+02:00, tomas@whalegate.ndb.mysql.com +1 -0
  Bug#29904 [Com]: Cluster replication breaks on large VARCHAR fields

Documented bugfix in mysql-5.1.22-ndb-6.2.5 changelog.

Left bug in PP status pending merge to mainline.