Bug #64336 Caused by error 2341: Internal program error
Submitted: 15 Feb 2012 5:30 Modified: 16 Sep 2016 14:16
Reporter: w yj Email Updates:
Status: Can't repeat Impact on me:
None 
Category:MySQL Cluster: Cluster (NDB) storage engine Severity:S1 (Critical)
Version:5.1.56-7.1.19 OS:Linux (2.6.32-100.34.1.el6uek.x86_64)
Assigned to: MySQL Verification Team CPU Architecture:Any

[15 Feb 2012 5:30] w yj 
Description:
node 4
RESTORE table: 7 3 rows applied
RedoPageCache: avoided 0 (0/0) page-reads
RedoOpenFileCache: Avoided 0 file-open/close closed: 0
Dbspj::execSTTOR() inst:0 phase=4
2012-02-15 20:51:04 [ndbd] INFO     -- Start phase 4 completed
2012-02-15 20:51:06 [ndbd] INFO     -- backup/Backup.cpp
2012-02-15 20:51:06 [ndbd] INFO     -- BACKUP (Line: 2926) 0x00000002
2012-02-15 20:51:06 [ndbd] INFO     -- Error handler shutting down system
2012-02-15 20:51:06 [ndbd] INFO     -- Error handler shutdown completed - exiting
2012-02-15 20:51:07 [ndbd] ALERT    -- Node 4: Forced node shutdown completed. Occured during startphase 5. Caused by error 2341: 'Internal program error (failed ndbrequire)(Internal error, programming error or missing error message, please report a bug). Temporary error, restart node'.

node 3
2012-02-15 20:51:04 [ndbd] INFO     -- timerHandlingLab now: 11487384 sent: 11487170 diff: 214
RedoPageCache: avoided 0 (0/0) page-reads
RedoOpenFileCache: Avoided 0 file-open/close closed: 0
Dbspj::execSTTOR() inst:0 phase=4
2012-02-15 20:51:04 [ndbd] INFO     -- Start phase 4 completed
2012-02-15 20:51:06 [ndbd] INFO     -- backup/Backup.cpp
2012-02-15 20:51:06 [ndbd] INFO     -- BACKUP (Line: 2926) 0x00000002
2012-02-15 20:51:06 [ndbd] INFO     -- Error handler shutting down system
2012-02-15 20:51:06 [ndbd] INFO     -- Error handler shutdown completed - exiting
2012-02-15 20:51:07 [ndbd] ALERT    -- Node 3: Forced node shutdown completed. Occured during startphase 5. Caused by error 2341: 'Internal program error (failed ndbrequire)(Internal error, programming error or missing error message, please report a bug). Temporary error, restart node'.

ndb_1_cluster.log 
2012-02-15 20:51:04 [MgmtSrvr] INFO     -- Node 4: Start phase 4 completed (system restart)
2012-02-15 20:51:04 [MgmtSrvr] INFO     -- Node 3: LQH: Rebuild ordered indexes complete
2012-02-15 20:51:04 [MgmtSrvr] INFO     -- Node 3: Start phase 4 completed (system restart)
2012-02-15 20:51:04 [MgmtSrvr] INFO     -- Node 3: GCP Monitor: Computed max GCP_SAVE lag to 131 seconds
2012-02-15 20:51:04 [MgmtSrvr] INFO     -- Node 3: GCP Monitor: Computed max GCP_COMMIT lag to 13 seconds
2012-02-15 20:51:05 [MgmtSrvr] INFO     -- Node 3: Local checkpoint 6 started. Keep GCI = 2283 oldest restorable GCI = 2283
2012-02-15 20:51:07 [MgmtSrvr] ALERT    -- Node 4: Forced node shutdown completed. Occured during startphase 5. Caused by error 2341: 'Internal program error (failed ndbrequire)(Internal error, programming error or missing error message, please report a bug). Temporary error, restart node'.
2012-02-15 20:51:07 [MgmtSrvr] ALERT    -- Node 3: Forced node shutdown completed. Occured during startphase 5. Caused by error 2341: 'Internal program error (failed ndbrequire)(Internal error, programming error or missing error message, please report a bug). Temporary error, restart node'.
2012-02-15 20:51:07 [MgmtSrvr] ALERT    -- Node 1: Node 3 Disconnected
2012-02-15 20:51:07 [MgmtSrvr] ALERT    -- Node 1: Node 4 Disconnected

How to repeat:
I don't know.i am just reboot the machine.
[15 Feb 2012 5:34] w yj 
all the log files

Attachment: log.zip (application/x-zip-compressed, text), 111.94 KiB.
[16 Feb 2012 17:15] Jonas Oreland 
this looks like either disk-full, file-system read-only
  or file permission problem.

if you upload a tracefile too,
maybe I can give more details

(and config.ini wouldn't hurt)

/Jonas
[21 Feb 2012 9:18] w yj 
------how can i find the trace files?

------disks are enough  space.--------------------------
[mysqlc@localhost mysqld_data]$ df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/VolGroup-lv_root
                       19G  3.5G   14G  20% /
tmpfs                 244M     0  244M   0% /dev/shm
/dev/sda1             485M   48M  413M  11% /boot
[mysqlc@localhost mysqld_data]$ 

------the file system permission is looks no problem-----
[21 Feb 2012 9:20] w yj 
the file system permission is looks no problem.

Attachment: file-system permission.txt (text/plain), 11.15 KiB.
[7 Sep 2012 6:42] chun man 
test
[21 Dec 2012 15:23] Mark Norman 
Any progress on this? I am experiencing it also.
[16 Sep 2016 14:16] MySQL Verification Team 
not enough data to go on, no way to reproduce it, looks like a filesystem error