Bug #27437 NDB Data node crashes during mysqldump inserts
Submitted: 26 Mar 2007 10:10 Modified: 15 May 2007 6:40
Reporter: Andrew Hutchings Email Updates:
Status: Closed Impact on me:
None 
Category:MySQL Cluster: Cluster (NDB) storage engine Severity:S2 (Serious)
Version:5.0.37 OS:Any (CentsOS 4.4)
Assigned to: Jonas Oreland CPU Architecture:Any
Tags: crash, ndb

[26 Mar 2007 10:10] Andrew Hutchings
Description:
Every hour a script takes a mysqldump from our production MySQL cluster (around 20meg of data) and inserts it into our development box (a single server MySQL cluster).  Every 7-9 days the data node on the development box crashes with the following error:

Time: Saturday 17 Mars 2007 - 05:46:10
Status: Temporary error, restart node
Message: Internal program error (failed ndbrequire) (Internal error,
programming error or missing error message, please report a bug)
Error: 2341
Error data: DblqhMain.cpp
Error object: DBLQH (Line: 2797) 0x0000000a
Program: /usr/sbin/ndbd
Pid: 28028
Trace: /var/lib/mysql-cluster/ndb_2_trace.log.3
Version: Version 5.0.37
***EOM***

It happened on 5.0.27 as well:

Time: Thursday 8 Mars 2007 - 17:46:08
Status: Temporary error, restart node
Message: Internal program error (failed ndbrequire) (Internal error,
programming error or missing error message, please report a bug)
Error: 2341
Error data: DblqhMain.cpp
Error object: DBLQH (Line: 2782) 0x0000000a
Program: /usr/sbin/ndbd
Pid: 30029
Trace: /var/lib/mysql-cluster/ndb_2_trace.log.2
Version: Version 5.0.27
***EOM***

How to repeat:
Insert data from a mysqldump every hour for a week.
[26 Mar 2007 10:12] Andrew Hutchings
ndb_error_reporter output

Attachment: ndb_error_report_20070326105627.tar.bz2 (application/octet-stream, text), 225.29 KiB.

[26 Mar 2007 10:36] Jonas Oreland
Seems to be a problem with TUPKEYREF during sync/start of LCP
(i.e 4.1/5.0 only)

Reason for TUPKEYREF seems to be auto-increment related
(not a bug per se...)

/Jonas
[26 Mar 2007 10:47] Andrew Hutchings
Digging through the tables I have found a couple of tables that have a primary key that isn't an auto_increment (I will kick someone's ass and get them changed shortly).  Could this be related?

If so then shouldn't their be a cleaner way of handling this?
[26 Mar 2007 10:54] Jonas Oreland
Hi,

>If so then shouldn't their be a cleaner way of handling this?
It's is a bug...

My notes was mainly notes for further digging...

/Jonas
[27 Mar 2007 6:10] Andrew Hutchings
Crashed again this morning, data attached, hope its useful.

Attachment: crash_20070327.tar.bz2 (application/octet-stream, text), 68.50 KiB.

[7 May 2007 6:52] Jonas Oreland
http://bugs.mysql.com/bug.php?id=28231
[7 May 2007 13:36] Jonas Oreland
http://bugs.mysql.com/bug.php?id=25493
[8 May 2007 5:49] Bugs System
A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/26266

ChangeSet@1.2305, 2007-05-08 07:49:05+02:00, jonas@perch.ndb.mysql.com +5 -0
  ndb - bug#27437
    bug in interpretedupdate (used for auto-increment) that made TUPKEYREF be sent twice
    this was however "normally" shadowed, but made LQH crash when LCP was about to start
[8 May 2007 7:33] Andrew Hutchings
Thanks Jonas,

I will apply it on the affected box at some point over the next couple of days and let you know if it happens again.
[8 May 2007 10:53] Bugs System
A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/26284

ChangeSet@1.2306, 2007-05-08 12:53:12+02:00, jonas@perch.ndb.mysql.com +1 -0
  ndb - bug#27437
    redo extra verification code so that tupkeyref is reset just before tupkeyreq
[8 May 2007 10:58] Jonas Oreland
Hi Andrew

FYI, the "extra" patch that I just committed is only needed if you're using
  a debug compiled ndbd.

/Jonas
[10 May 2007 12:56] Bugs System
Pushed into 5.0.42
[10 May 2007 12:56] Bugs System
Pushed into 5.1.19-beta
[15 May 2007 6:40] Jon Stephens
Thank you for your bug report. This issue has been committed to our source repository of that product and will be incorporated into the next release.

If necessary, you can access the source repository and build the latest available version, including the bug fix. More information about accessing the source trees is available at

    http://dev.mysql.com/doc/en/installing-source.html

Documented bugfix in 5.0.42 and 5.1.19 changelogs.