MySQL Bugs: #48033: GCP stop under virtually no load, followed by full cluster crash

Bug #48033	GCP stop under virtually no load, followed by full cluster crash
Submitted:	14 Oct 2009 8:05	Modified:	7 Mar 2016 6:36
Reporter:	Daniel Herlitz	Email Updates:
Status:	Can't repeat	Impact on me:	None
Category:	MySQL Cluster: Cluster (NDB) storage engine	Severity:	S2 (Serious)
Version:	mysql-5.1-telco-7.0	OS:	Linux
Assigned to:	MySQL Verification Team	CPU Architecture:	Any
Tags:	7.0.7

Description:
After running fine for a week under very low load both NDB nodes suddendly crashed this morning. 

Node 3 killed this node because GCP stop was detected

Forced node shutdown completed. Caused by error 2303

I will attach all relevant log parts, ndb_error_report and config.ini

How to repeat:
No idea

Suggested fix:
No idea

Logs, config files

Attachment: 2009-10-14.tar.gz (application/x-gzip, text), 174.72 KiB.

I don't know if this is related, but when we try to start NDB, both nodes kind of seem locked up in weird way, printing about 100 lines / second to the log log files of this message:

delay: reqs=348

swap ?
other processes on machines ?

LockPagesInMemory (requires root) or RealtimeScheduler (requires root)
Can make ndbd more robust in competing for hw-resources

The three machines are dedicated (though virtual) to running the MySQL cluster (no other services running on them). They are part of a staging environment which noone was using at the time of the crash (more or less zero load).

Daniel,

Running cluster on virtual machines isn't a very good idea. But does the virtual machines have dedicated RAM or do they share them with other virtual machines? In that case you "RAM" might have ended up on disk anyway. What software do you use for virtualization?

/Gustaf

Those machines have 3 GBs of dedicated memory (no "overallocation"). We run Oracle VM (Xen) for virtualization.

- after few years: not reproducible on any of the current 7.3/7.4 releases

kind regards
Bogdan Kecman