Bug #54586 ndb node crash during backup
Submitted: 17 Jun 2010 14:49 Modified: 8 Sep 2016 5:40
Reporter: Casey Brown Email Updates:
Status: Can't repeat Impact on me:
None 
Category:MySQL Cluster: Cluster (NDB) storage engine Severity:S2 (Serious)
Version:mysql-5.1-telco-7.1 OS:Linux
Assigned to: CPU Architecture:Any
Tags: mysql-5.1.44 ndb-7.1.3

[17 Jun 2010 14:49] Casey Brown
Description:
2 datanodes, one crashes consistently during backup.

I presume it has something to do with my data, as backups have been working fine since our upgrade to 7.1.3

This dataset makes heavy use of blobs stored as diskdata.

Status: Temporary error, restart node
Message: Internal program error (failed ndbrequire) (Internal error, programming error or missing error message, please report a bug)
Error: 2341
Error data: dbtup/DbtupPageMap.cpp
Error object: DBTUP (Line: 103) 0x0000000a
Program: /usr/sbin/ndbd
Pid: 7103
Version: mysql-5.1.44 ndb-7.1.3
Trace: /var/lib/mysql/data/ndb_2_trace.log.6

ndb_error_reporter log attached.

How to repeat:
start backup N
[18 Jun 2010 11:08] Jonas Oreland
error reporter is empty...
[18 Jun 2010 19:24] Casey Brown
Not sure what happened there.

Let me know if you need more than just the trace file.
[21 Jun 2010 11:41] Hartmut Holzgraefe
failure is in:

 95 Uint32 Dbtup::getRealpid(Fragrecord* regFragPtr, Uint32 logicalPageId)
 96 {
 97   DynArr256 map(c_page_map_pool, regFragPtr->m_page_map);
 98   Uint32 * ptr = map.get(2 * logicalPageId);
 99   if (likely(ptr != 0))
100   {
101     return * ptr;
102   }
103   ndbrequire(false);
104   return RNIL;
105 }
[23 Jun 2010 22:08] Casey Brown
I tracked the root of this issue down to a single record, which had a bad reference to a hidden blob table. (varchar(512))
From previous experience, I assume the data in that column is lost.

Only known workaround:
Manually dump the data for that table around the record (mysqldump --where='id<>badrecord')
Reinsert the data in a temp table
Rename the original table to _bak
Rename temp to original
INSERT INTO original (SELECT col1, col2, ... FROM _bak WHERE id = badrecord)
(basically list all the columns except the known bad blob column)
drop table _bak

System now runs regular backups.
[6 Jul 2010 9:53] Andrew Hutchings
Hello Casey,

Varchar(512) will not have a hidden blob table.  Only Text and Blob columns will.
[6 Jul 2010 15:37] Casey Brown
I was under the impression that for VARCHAR, anything over 255 bytes was stored in the hidden blob tables.

Re-reading the docs, I am unsure where I came by that impression.
On a tangent, looks like there is a performance benefit to using VARCHAR instead of TEXT if you can stay under the 8k row limit.

However, the issue still stands. The behavior is very similar to the blob table hole issue, in that you can select around the hole.
[10 Jul 2010 10:55] Jonas Oreland
this could maybe be http://bugs.mysql.com/bug.php?id=54986
[8 Sep 2016 5:40] MySQL Verification Team
not a repeatable bug, possibly fixed by 54986