Bug #36204 Error 2306 & 2341 after restarting mysqld (API node) on cluster
Submitted: 18 Apr 2008 10:12 Modified: 7 Mar 2016 6:30
Reporter: matteo brancaleoni Email Updates:
Status: Can't repeat Impact on me:
None 
Category:MySQL Cluster: Cluster (NDB) storage engine Severity:S3 (Non-critical)
Version:mysql-5.1 OS:Linux (Centos 5)
Assigned to: Bogdan Kecman CPU Architecture:Any
Tags: 5.1.23-rc
Triage: Triaged: D2 (Serious) / R6 (Needs Assessment) / E6 (Needs Assessment)

[18 Apr 2008 10:12] matteo brancaleoni
Description:

After a couple of days running the cluster, today I restarted 1 mysqld node.
when back online, it was unable to access all the tables of the db with these 
warnings:
mysql> show warnings;
+-------+------+--------------------------------------------------------------------------------------------------------------+
| Level | Code | 
Message                                                                                                      |
+-------+------+--------------------------------------------------------------------------------------------------------------+
| Error | 1296 | Got error 4243 'Index not found' from 
NDB                                                                    |
| Error | 1296 | Got error 284 'Table not defined in transaction coordinator' 
from Could not log query '%s' on other mysqld's |
| Error | 1296 | Got error 4243 'Index not found' from 
NDB                                                                    |
| Error | 1296 | Got error 4243 'Index not found' from 
NDB                                                                    |
| Error | 1296 | Got error 284 'Table not defined in transaction coordinator' 
from Could not log query '%s' on other mysqld's |
| Error | 1296 | Got error 4243 'Index not found' from 
NDB                                                                    |
| Error | 1296 | Got error 4243 'Index not found' from 
NDB                                                                    |
| Error | 1296 | Got error 284 'Table not defined in transaction coordinator' 
from Could not log query '%s' on other mysqld's |
| Error | 1296 | Got error 4243 'Index not found' from 
NDB                                                                    |
+-------+------+--------------------------------------------------------------------------------------------------------------+

After that, I decided to stop the API node, remove the DB from disk 
(/var/lib/mysql/db_name) and restart the API node.
After that, i recreated the db, in order to populate it from ndb, but these 
errors appeared:
mysql> show warnings;
+---------+------+----------------------------------------------------+
| Level   | Code | Message                                            |
+---------+------+----------------------------------------------------+
| Warning | 1050 | Local table openser.aliases shadows ndb table      |
| Warning | 1050 | Local table openser.dispatcher shadows ndb table   |
| Warning | 1050 | Local table openser.gw_grp shadows ndb table       |
| Warning | 1050 | Local table openser.imc_members shadows ndb table  |
| Warning | 1050 | Local table openser.lcr shadows ndb table          |
| Warning | 1050 | Local table openser.location shadows ndb table     |
| Warning | 1050 | Local table openser.missed_calls shadows ndb table |
| Warning | 1050 | Local table openser.re_grp shadows ndb table       |
| Warning | 1050 | Local table openser.trusted shadows ndb table      |
+---------+------+----------------------------------------------------+

Finally, as last resort, I shutted down the cluster with SHUTDOWN from 
mgm, then restarted back.

But when trying to restart the data nodes, I get these different errors:

Node 2: Forced node shutdown completed. Occured during startphase 4. Caused by 
error 2306: 'Pointer too large(Internal error, programming error or missing 
error message, please report a bug). Temporary error, restart node

Node 3: Forced node shutdown completed. Occured during startphase 4. Caused by 
error 2341: 'Internal program error (failed ndbrequire)(Internal error, 
programming error or missing error message, please report a bug). Temporary 
error, restart node

How to repeat:
still trying to repeat with same environment.
[18 Apr 2008 10:18] matteo brancaleoni
ndb logs of first data node

Attachment: node_1_logs.tgz (application/x-compressed-tar, text), 171.71 KiB.

[18 Apr 2008 10:19] matteo brancaleoni
ndb logs of second data node

Attachment: node_2_logs.tgz (application/x-compressed-tar, text), 214.33 KiB.

[18 Apr 2008 13:41] matteo brancaleoni
I was able to reproduce it. 

Same error if I try to restart 1 data node.

I've the full data dir with logs, 36megs tgz.
You can find it here:
http://www.matteobrancaleoni.net/node_2_full_data_dir_with_logs.tar.gz
[26 Sep 2008 12:24] Elbert Joseph Layco
I have the same problem. I already reported this bug but for version mysql-5.0.67-ndb-6.2.15.
[18 May 2009 13:56] Jonathan Miller
Pekka, do you need more feedback on this? Per Martin S.
[7 Mar 2016 6:30] Bogdan Kecman
after few years - not reproducible on any of the recent 7.3/7.4 releases

Bogdan Kecman