MySQL Bugs: #75527: Forced node shutdown completed...Initiated by signal 11.

Bug #75527	Forced node shutdown completed...Initiated by signal 11.
Submitted:	16 Jan 2015 11:49	Modified:	22 Jul 2016 13:12
Reporter:	Marek Knappe	Email Updates:
Status:	Can't repeat	Impact on me:	None
Category:	MySQL Cluster: Cluster (NDB) storage engine	Severity:	S1 (Critical)
Version:	mysql-5.6.21 ndb-7.3.7	OS:	Linux (CentOS 5.8)
Assigned to:	Bogdan Kecman	CPU Architecture:	Any
Tags:	ndb;signal 11;crash;mysq;node shutdown

Description:
Hi,
I have faced cluster shutdown after executing following queries:

mysql> select * from ec_dump2 into outfile '/data/backup/tmp/ec_dump2.sql';
Query OK, 4248932 rows affected (16 min 56.58 sec)
mysql> truncate ec_dump2;
ERROR 1296 (HY000): Got error 157 'Unknown error code' from NDBCLUSTER

From log I could observe this :
2015-01-16 19:20:21 [MgmtSrvr] INFO     -- Node 5: Event buffer status: used=77KB(0%) alloc=16040KB(0%) max=0B apply_epoch=34523303/4 latest_epoch=34523303/18
2015-01-16 19:27:38 [MgmtSrvr] ALERT    -- Node 3: Forced node shutdown completed. Occured during startphase 0. Initiated by signal 11.
2015-01-16 19:27:38 [MgmtSrvr] ALERT    -- Node 1: Node 3 Disconnected
2015-01-16 19:27:38 [MgmtSrvr] ALERT    -- Node 2: Forced node shutdown completed. Occured during startphase 0. Initiated by signal 11.
2015-01-16 19:27:38 [MgmtSrvr] ALERT    -- Node 1: Node 2 Disconnected

Here are necessary logs from bot nodes:
https://www.dropbox.com/s/z2v483k51p4p9oq/ndb_error_report_20150116222429.tar.bz2?dl=0

Please note time diff between bot data nodes:
[root@yt-db001 ~]# date
Fri Jan 16 21:50:02 EST 2015
[root@vt-db002 ~]# date
Fri Jan 16 21:56:53 EST 2015

[root@vm-ytlb001 mysql-cluster]# ndb_mgm -e show
Connected to Management Server at: localhost:1186
Cluster Configuration
---------------------
[ndbd(NDB)]     2 node(s)
id=2    @192.168.3.11  (mysql-5.6.21 ndb-7.3.7, Nodegroup: 0, *)
id=3    @192.168.3.12  (mysql-5.6.21 ndb-7.3.7, Nodegroup: 0)

[ndb_mgmd(MGM)] 1 node(s)
id=1    @192.168.3.20  (mysql-5.6.21 ndb-7.3.7)

[mysqld(API)]   2 node(s)
id=4    @192.168.3.20  (mysql-5.6.21 ndb-7.3.7)
id=5    @192.168.3.22  (mysql-5.6.21 ndb-7.3.7)

Please note that after cluster shutdown I have started cluster from mysql user and you could probably see that in logs also. Had to switch for root as it was started from root previously.

How to repeat:
Can't test it as this cluster is in "production".

I would like to add also that on this cluster we are unable to perform any backup which is also critical for us:

ndb_mgm> START BACKUP WAIT COMPLETED
Connected to Management Server at: localhost:1186
Waiting for completed, this may take several minutes
Node 2: Backup 11 started from 1 has been aborted. Error: 4237
Backup failed
*  3001: Could not start backup
*        Too many triggers: Permanent error: Application error

Hi Umesh,
Any progress on this?

Sorry, I'm not able to reproduce this issue at my end.
I'll check internally if this has been fixed recently in 7.3.9 builds and then I would update you this.

Thanks for Info. Looking forward for more news.

Hi,

reviewing this bug after some time passed.

I cannot reproduce this on 
 - 7.0.37
 - 7.1.34
 - 7.3.13
 - 7.4.11

Did you upgraded your system maybe recently and did you encountered this same bug on the new release? I'm unable to grab your log from dropbox so if you did encounter this same error on one of the releases I can't reproduce it on please upload the full logs again

kind regards
Bogdan Kecman