MySQL Bugs: #27437: NDB Data node crashes during mysqldump inserts

Bug #27437	NDB Data node crashes during mysqldump inserts
Submitted:	26 Mar 2007 10:10	Modified:	15 May 2007 6:40
Reporter:	Andrew Hutchings	Email Updates:
Status:	Closed	Impact on me:	None
Category:	MySQL Cluster: Cluster (NDB) storage engine	Severity:	S2 (Serious)
Version:	5.0.37	OS:	Any (CentsOS 4.4)
Assigned to:	Jonas Oreland	CPU Architecture:	Any
Tags:	crash, ndb

Description:
Every hour a script takes a mysqldump from our production MySQL cluster (around 20meg of data) and inserts it into our development box (a single server MySQL cluster).  Every 7-9 days the data node on the development box crashes with the following error:

Time: Saturday 17 Mars 2007 - 05:46:10
Status: Temporary error, restart node
Message: Internal program error (failed ndbrequire) (Internal error,
programming error or missing error message, please report a bug)
Error: 2341
Error data: DblqhMain.cpp
Error object: DBLQH (Line: 2797) 0x0000000a
Program: /usr/sbin/ndbd
Pid: 28028
Trace: /var/lib/mysql-cluster/ndb_2_trace.log.3
Version: Version 5.0.37
***EOM***

It happened on 5.0.27 as well:

Time: Thursday 8 Mars 2007 - 17:46:08
Status: Temporary error, restart node
Message: Internal program error (failed ndbrequire) (Internal error,
programming error or missing error message, please report a bug)
Error: 2341
Error data: DblqhMain.cpp
Error object: DBLQH (Line: 2782) 0x0000000a
Program: /usr/sbin/ndbd
Pid: 30029
Trace: /var/lib/mysql-cluster/ndb_2_trace.log.2
Version: Version 5.0.27
***EOM***

How to repeat:
Insert data from a mysqldump every hour for a week.

ndb_error_reporter output

Attachment: ndb_error_report_20070326105627.tar.bz2 (application/octet-stream, text), 225.29 KiB.

Seems to be a problem with TUPKEYREF during sync/start of LCP
(i.e 4.1/5.0 only)

Reason for TUPKEYREF seems to be auto-increment related
(not a bug per se...)

/Jonas

Digging through the tables I have found a couple of tables that have a primary key that isn't an auto_increment (I will kick someone's ass and get them changed shortly).  Could this be related?

If so then shouldn't their be a cleaner way of handling this?

Hi,

>If so then shouldn't their be a cleaner way of handling this?
It's is a bug...

My notes was mainly notes for further digging...

/Jonas

Crashed again this morning, data attached, hope its useful.

Attachment: crash_20070327.tar.bz2 (application/octet-stream, text), 68.50 KiB.

http://bugs.mysql.com/bug.php?id=28231

http://bugs.mysql.com/bug.php?id=25493

A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/26266

ChangeSet@1.2305, 2007-05-08 07:49:05+02:00, jonas@perch.ndb.mysql.com +5 -0
  ndb - bug#27437
    bug in interpretedupdate (used for auto-increment) that made TUPKEYREF be sent twice
    this was however "normally" shadowed, but made LQH crash when LCP was about to start

Thanks Jonas,

I will apply it on the affected box at some point over the next couple of days and let you know if it happens again.

A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/26284

ChangeSet@1.2306, 2007-05-08 12:53:12+02:00, jonas@perch.ndb.mysql.com +1 -0
  ndb - bug#27437
    redo extra verification code so that tupkeyref is reset just before tupkeyreq

Hi Andrew

FYI, the "extra" patch that I just committed is only needed if you're using
  a debug compiled ndbd.

/Jonas

Pushed into 5.0.42

Pushed into 5.1.19-beta

Thank you for your bug report. This issue has been committed to our source repository of that product and will be incorporated into the next release.

If necessary, you can access the source repository and build the latest available version, including the bug fix. More information about accessing the source trees is available at

    http://dev.mysql.com/doc/en/installing-source.html

Documented bugfix in 5.0.42 and 5.1.19 changelogs.