Bug #26386 ndbd wont start after changing schema
Submitted: 14 Feb 2007 23:22 Modified: 28 Feb 2008 11:20
Reporter: Cooper Quintin Email Updates:
Status: No Feedback Impact on me:
None 
Category:MySQL Cluster: Cluster (NDB) storage engine Severity:S2 (Serious)
Version:5.0.33 OS:Linux (FC4)
Assigned to: CPU Architecture:Any
Tags: hh_monday

[14 Feb 2007 23:22] Cooper Quintin
Description:
I added a table to one of my databases after stopping ndbd and mysqld on
on one of the nodes.  Afterwards on one of the nodes, sqld was started and put in background and then ndbd was started.  Ndbd failed to start.  Tried again to start and still would not start

Out.log
----------------
2007-02-12 06:06:28 [ndbd] INFO     -- Angel pid: 26402 ndb pid: 26403
2007-02-12 06:06:28 [ndbd] INFO     -- NDB Cluster -- DB node 3
2007-02-12 06:06:28 [ndbd] INFO     -- Version 5.0.33 --
2007-02-12 06:06:28 [ndbd] INFO     -- Configuration fetched at 192.168.1.200 port 1186
2007-02-12 06:06:28 [ndbd] INFO     -- Start initiated (version 5.0.33)
2007-02-12 06:06:29 [ndbd] INFO     -- Error handler startup shutting down system
2007-02-12 06:06:29 [ndbd] INFO     -- Error handler shutdown completed - exiting
2007-02-12 06:06:29 [ndbd] INFO     -- Angel received ndbd startup failure count 1.
2007-02-12 06:06:29 [ndbd] ALERT    -- Node 3: Forced node shutdown completed. Occured during startphase 2. Initiated by signal 0. Caused by error 2341: 'Internal program error (failed ndbrequire)(Internal error, programming error or missing error message, please report a bug). Temporary error

error.log
--------------------------
Time: Monday 12 February 2007 - 06:06:29
Status: Temporary error, restart node
Message: Internal program error (failed ndbrequire) (Internal error, programming error or missing error message, please report a bug)
Error: 2341
Error data: Dbdict.cpp
Error object: DBDICT (Line: 1080) 0x0000000a
Program: /home/mysql3/mysql/libexec/ndbd
Pid: 26403
Trace: /home/mysql3/mysql-max-5.0.33-linux-i686/data/ndb_3_trace.log.1
Version: Version 5.0.33
***EOM***

trace.log
----------------
attached

How to repeat:
stop one ndbd node and one mysqld node on the same server.  on another ndbd node change the schema of a database by adding a table.  Then start mysqld, then try to start ndbd

Suggested fix:
start ndbd with --initial option
[14 Feb 2007 23:42] Cooper Quintin
*** I just realized that it was not an add table I did but an ALTER ADD column
[14 May 2007 19:08] Tomas Ulin
Cooper,

so let me repeat to see if I've understood.

1. You have a 2-node cluster up and running with a schema
2. you bring down one node
3. you do an alter table add column
4. you attempt to restart the node, but it fails

if so a "similar bug" has been fixed in a later release.

I'm afraid we cannot do more with this bug without either a filesystem which causes the problem, or a way to reliable reproduce this.

It either of this something that you are able to provide us with?

BR,

Tomas
[14 May 2007 22:18] Tomas Ulin
seems to be the same 27952
[15 May 2007 6:44] Bugs System
A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/26663

ChangeSet@1.2487, 2007-05-15 09:03:00+02:00, tomas@whalegate.ndb.mysql.com +3 -0
  Bug #26386 ndbd wont start after changing schema
      backported error code from 5.1
      split error check when reading schema file
[23 May 2007 8:24] Bugs System
Pushed into 5.1.19-beta
[23 May 2007 8:24] Bugs System
Pushed into 5.0.44
[14 Jun 2007 23:00] Bugs System
No feedback was provided for this bug for over a month, so it is
being suspended automatically. If you are able to provide the
information that was originally requested, please do so and change
the status of the bug back to "Open".
[28 Jan 2008 11:20] Valeriy Kravchuk
Is this bug fixed? Shouldn't it be closed then?
[29 Feb 2008 0:00] Bugs System
No feedback was provided for this bug for over a month, so it is
being suspended automatically. If you are able to provide the
information that was originally requested, please do so and change
the status of the bug back to "Open".