MySQL Bugs: #65361: ndbmtd restart (non initial) impossible after upgrading 7.15a -> 7.2.5

Bug #65361	ndbmtd restart (non initial) impossible after upgrading 7.15a -> 7.2.5
Submitted:	18 May 2012 14:26	Modified:	22 Aug 2016 13:05
Reporter:	Marco Sperandio	Email Updates:
Status:	No Feedback	Impact on me:	None
Category:	MySQL Cluster: Cluster (NDB) storage engine	Severity:	S1 (Critical)
Version:	7.2.6	OS:	Linux (Rhel 5.4 64 bit)
Assigned to:	MySQL Verification Team	CPU Architecture:	Any

Description:
Hello,

the last week we tested the upgrade from MySQL Cluster 7.1.15a to MySQL Cluster 7.2.5

With an empty database the upgrade ran smooth, the datanodes came up without issues.

With our corporate Database (around 35 GB DB) on the same MySQL Cluster configuration, after the successful installation of the RPM's, the startup procedure of ndbdmtd remain stuck in the RestartCreateObject procedure, with the master datanode at 99.8% ndbmtd cpu usage and 1.00 15min load. (OS: RHEL 5.4 64bit)

Every time We tried to start the datanodes (1 ore 2 Datanodes makes no difference), the process frozen at RestartCreateObject(52).

We have also maximised the logging verbosity (15):

LogLevelCheckpoint=15
LogLevelError=15
LogLevelInfo=15
LogLevelConnection=15
LogLevelStatistic=15

We tried to start datanodes with --core-file option TRUE, and -V option enabled to reach the maximum verbosity but the startup procedure simply got frozen after some minutes, with no chanches to start the datanodes (no progress after 9-10 hours).

No errors shown in ndb_x_out.log of datanodes and ndb_x_cluster.log of management nodes.

It seems that something in the schema of our Database break off the startup procedure.

Someone encountered the same problem?

Thank You in advance

Marco Sperandio

How to repeat:
I can't supply mode details because I don't know how RestartCreateObject(52) is related to a schema object or a phisical data.

Moved to MySQL cluster generic.

Please take note that the problem is still present in 7.2.6

ndbmtd 7.2.6 freezes during "restartCreateObj(52) file: 1"
I think it could be a problem loading DB schema, but with no additional informations by the daemon is impossible for me to find the exact table/row/index that cause the problem.

I hope to hear notices soon, 

Thank You in advance

Best regards

Marco Sperandio

Hi Marco,

Could you please attach the cluster logs? Preferably using the ndb_error_reporter utility:

  http://dev.mysql.com/doc/refman/5.5/en/mysql-cluster-programs-ndb-error-reporter.html

Regards,
Gustaf

Hello Gustaf,

I've uploaded the requested log to You the 25 May, have You some news for me?
thank You in advance

Best regards.

Marco Sperandio

Hi,
reviewing this after few years, the log does not show much but it does resemble a SUMA bug that was fixed in 2015. Not 100% what version the fix was introduced but it's for sure in 7.2.23+

can you let us know if you are having this issue with latest 7.2.24 or 7.4.11

best regards
Bogdan Kecman

No feedback was provided for this bug for over a month, so it is
being suspended automatically. If you are able to provide the
information that was originally requested, please do so and change
the status of the bug back to "Open".