MySQL Bugs: #85901: Starting MySQL. ERROR! The server quit without updating PID file (/var/lib/mysq

Bug #85901	Starting MySQL. ERROR! The server quit without updating PID file (/var/lib/mysq
Submitted:	11 Apr 2017 12:42	Modified:	13 Apr 2017 4:59
Reporter:	Arun Kumar Rajagopal	Email Updates:
Status:	Duplicate	Impact on me:	None
Category:	MySQL Cluster Manager	Severity:	S1 (Critical)
Version:	5.6.28-ndb-7.4.10	OS:	CentOS (ERROR! MySQL server PID file could not be found!)
Assigned to:		CPU Architecture:	Any
Tags:	#Centos7.3 #mysqlcluster

Description:
Scenario:

While setting up the mysql cluster in NDB engine , Couple of time the API node went "mysql" service not starting , It seems the below issue was repeating again and again , 

Error : 1 
ERROR! MySQL server PID file could not be found!

Error : 2
Starting MySQL. ERROR! The server quit without updating PID file (/var/lib/mysql/centosvm02.pid).

Error : 3 (logs)
Configuration error: Error: Could not alloc node id at 10.0.0.4 port 1186: Connection done from wrong host ip 10.0.0.10.
2017-04-11 17:55:48 9572 [ERROR] NDB: error (1) Configuration error: Error: Could not alloc node id at 10.0.0.4 port 1186: Connection done from wrong host ip 10.0.0.10.
2017-04-11 17:55:48 9572 [ERROR] NDB: Failed to initialize connection(s)
2017-04-11 17:55:48 9572 [ERROR] NDB: Failed to initialize ndbcluster, aborting!
2017-04-11 17:55:48 9572 [ERROR] NDB: Use --skip-ndbcluster to start without ndbcluster.
170411 17:55:48 mysqld_safe mysqld from pid file /var/lib/mysql/centosvm02.pid ended
170411 17:58:06 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
2017-04-11 17:58:06 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestamp server option (see documentation for more details).
2017-04-11 17:58:06 0 [Note] /usr/sbin/mysqld (mysqld 5.6.28-ndb-7.4.10-cluster-gpl) starting as process 9815 ...
2017-04-11 17:58:06 9815 [Note] Plugin 'FEDERATED' is disabled.
2017-04-11 17:58:06 9815 [Note] InnoDB: Using atomics to ref count buffer pool pages
2017-04-11 17:58:06 9815 [Note] InnoDB: The InnoDB memory heap is disabled
2017-04-11 17:58:06 9815 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
2017-04-11 17:58:06 9815 [Note] InnoDB: Memory barrier is not used
2017-04-11 17:58:06 9815 [Note] InnoDB: Compressed tables use zlib 1.2.3
2017-04-11 17:58:06 9815 [Note] InnoDB: Using Linux native AIO
2017-04-11 17:58:06 9815 [Note] InnoDB: Using CPU crc32 instructions
2017-04-11 17:58:06 9815 [Note] InnoDB: Initializing buffer pool, size = 128.0M
2017-04-11 17:58:06 9815 [Note] InnoDB: Completed initialization of buffer pool
2017-04-11 17:58:06 9815 [Note] InnoDB: Highest supported file format is Barracuda.
2017-04-11 17:58:06 9815 [Note] InnoDB: The log sequence numbers 1626007 and 1626007 in ibdata files do not match the log sequence number 1626027 in the ib_logfiles!
2017-04-11 17:58:06 9815 [Note] InnoDB: Database was not shutdown normally!
2017-04-11 17:58:06 9815 [Note] InnoDB: Starting crash recovery.
2017-04-11 17:58:06 9815 [Note] InnoDB: Reading tablespace information from the .ibd files...
2017-04-11 17:58:06 9815 [Note] InnoDB: Restoring possible half-written data pages
2017-04-11 17:58:06 9815 [Note] InnoDB: from the doublewrite buffer...
2017-04-11 17:58:06 9815 [Note] InnoDB: 128 rollback segment(s) are active.
2017-04-11 17:58:06 9815 [Note] InnoDB: Waiting for purge to start
2017-04-11 17:58:06 9815 [Note] InnoDB: 5.6.28 started; log sequence number 1626027
2017-04-11 17:58:06 9815 [Note] NDB: Changed global value of binlog_format from STATEMENT to MIXED
Configuration error: Error: Could not alloc node id at 10.0.0.4 port 1186: Connection done from wrong host ip 10.0.0.10.
2017-04-11 17:58:06 9815 [ERROR] NDB: error (1) Configuration error: Error: Could not alloc node id at 10.0.0.4 port 1186: Connection done from wrong host ip 10.0.0.10.
2017-04-11 17:58:06 9815 [ERROR] NDB: Failed to initialize connection(s)
2017-04-11 17:58:06 9815 [ERROR] NDB: Failed to initialize ndbcluster, aborting!
2017-04-11 17:58:06 9815 [ERROR] NDB: Use --skip-ndbcluster to start without ndbcluster.
170411 17:58:06 mysqld_safe mysqld from pid file /var/lib/mysql/centosvm02.pid ended
170411 18:02:16 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
2017-04-11 18:02:17 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestamp server option (see documentation for more details).
2017-04-11 18:02:17 0 [Note] /usr/sbin/mysqld (mysqld 5.6.28-ndb-7.4.10-cluster-gpl) starting as process 10052 ...
2017-04-11 18:02:17 10052 [Note] Plugin 'FEDERATED' is disabled.
2017-04-11 18:02:17 10052 [Note] InnoDB: Using atomics to ref count buffer pool pages
2017-04-11 18:02:17 10052 [Note] InnoDB: The InnoDB memory heap is disabled
2017-04-11 18:02:17 10052 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
2017-04-11 18:02:17 10052 [Note] InnoDB: Memory barrier is not used
2017-04-11 18:02:17 10052 [Note] InnoDB: Compressed tables use zlib 1.2.3
2017-04-11 18:02:17 10052 [Note] InnoDB: Using Linux native AIO
2017-04-11 18:02:17 10052 [Note] InnoDB: Using CPU crc32 instructions
2017-04-11 18:02:17 10052 [Note] InnoDB: Initializing buffer pool, size = 128.0M
2017-04-11 18:02:17 10052 [Note] InnoDB: Completed initialization of buffer pool
2017-04-11 18:02:17 10052 [Note] InnoDB: Highest supported file format is Barracuda.
2017-04-11 18:02:17 10052 [Note] InnoDB: The log sequence numbers 1626007 and 1626007 in ibdata files do not match the log sequence number 1626027 in the ib_logfiles!
2017-04-11 18:02:17 10052 [Note] InnoDB: Database was not shutdown normally!
2017-04-11 18:02:17 10052 [Note] InnoDB: Starting crash recovery.
2017-04-11 18:02:17 10052 [Note] InnoDB: Reading tablespace information from the .ibd files...
2017-04-11 18:02:17 10052 [Note] InnoDB: Restoring possible half-written data pages
2017-04-11 18:02:17 10052 [Note] InnoDB: from the doublewrite buffer...
2017-04-11 18:02:17 10052 [Note] InnoDB: 128 rollback segment(s) are active.
2017-04-11 18:02:17 10052 [Note] InnoDB: Waiting for purge to start
2017-04-11 18:02:17 10052 [Note] InnoDB: 5.6.28 started; log sequence number 1626027
2017-04-11 18:02:17 10052 [Note] NDB: Changed global value of binlog_format from STATEMENT to MIXED
Configuration error: Error: Could not alloc node id at 10.0.0.4 port 1186: Connection done from wrong host ip 10.0.0.10.
2017-04-11 18:02:17 10052 [ERROR] NDB: error (1) Configuration error: Error: Could not alloc node id at 10.0.0.4 port 1186: Connection done from wrong host ip 10.0.0.10.
2017-04-11 18:02:17 10052 [ERROR] NDB: Failed to initialize connection(s)
2017-04-11 18:02:17 10052 [ERROR] NDB: Failed to initialize ndbcluster, aborting!
2017-04-11 18:02:17 10052 [ERROR] NDB: Use --skip-ndbcluster to start without ndbcluster.
170411 18:02:17 mysqld_safe mysqld from pid file /var/lib/mysql/centosvm02.pid ended

How to repeat:

Step 1: #mysqladmin -u root -p shutdown

Step 2: #mysqld_safe &

Step 3: # service mysql start

Error: 
ERROR! MySQL server PID file could not be found!

Hello Arun,

Thank you for the report.
The reported problem "The server quit without updating PID file" was fixed in 5.6.36 after Bug #84263. Marking this as duplicate of Bug #84263.
Also, other noted issue  "Error: Could not alloc node id at 10.0.0.4 port 1186: Connection done from wrong host ip 10.0.0.10." - From the error messages it seems to me that you are trying to connect a data node from 10.0.0.10 but you configured to only allow data nodes to connect from 10.0.0.4.

Thanks,
Umesh

Thanks Umesh..