Bug #30896 unwanted shutdown of ndb slave cluster
Submitted: 7 Sep 2007 14:25 Modified: 18 Sep 2007 10:00
Reporter: Nicolas Legrand Email Updates:
Status: Closed Impact on me:
None 
Category:MySQL Cluster: Replication Severity:S1 (Critical)
Version:5.1.20beta OS:Linux
Assigned to: CPU Architecture:Any
Tags: dbtup, ndb, replication, salve

[7 Sep 2007 14:25] Nicolas Legrand
Description:
When loading a massive insert on the Master cluster (1 ndb_mgm & 4 nodes), slave cluster (1 ndb_mgm & 2 nodes) are failling, and show slave status is reporting :
 
Last_SQL_Error: Error 'Can't lock file (errno: 157)' in Write_rows
event: when locking tables 

on one of the 2 nodes, I have the following error in the ndb_X_out.log:
Time: Friday 7 September 2007 - 10:53:25
Status: Temporary error, restart node
Message: Internal program error (failed ndbrequire) (Internal error, programming error or missing error message, please report a bug)
Error: 2341
Error data: dbtup/DbtupDiskAlloc.cpp
2007-09-07 10:53:25 [ndbd] INFO     -- dbtup/DbtupDiskAlloc.cpp
2007-09-07 10:53:25 [ndbd] INFO     -- DBTUP (Line: 625) 0x0000000e
2007-09-07 10:53:25 [ndbd] INFO     -- Error handler shutting down system
2007-09-07 10:53:26 [ndbd] INFO     -- Error handler shutdown completed - exiting
2007-09-07 10:53:26 [ndbd] ALERT    -- Node 3: Forced node shutdown completed. Caused by error 2341: 'Internal program error (failed ndbrequire)(Internal error,
programming error or missing error message, please report a bug). Temporary error, restart node'.

How to repeat:
Launching a massive Insert on Master cluster.
[8 Sep 2007 21:18] Hartmut Holzgraefe
To analyze this we need your config.ini and the complete set 
of node error and trace logs. You can either collect them 
manually from the cluster nodes or use the ndb_error_reporter
tool to do so:

http://dev.mysql.com/doc/refman/5.0/en/mysql-cluster-utilities-ndb-error-reporter.html
[18 Sep 2007 10:00] Nicolas Legrand
It seems that some differences between table on slave and replication might be the cause of that issue.