MySQL Bugs: #31666: Instance manager fails to stop instance and falsely reports successful shutdown

Bug #31666	Instance manager fails to stop instance and falsely reports successful shutdown
Submitted:	17 Oct 2007 14:09	Modified:	19 Nov 2007 10:09
Reporter:	Hans Habers	Email Updates:
Status:	No Feedback	Impact on me:	None
Category:	Instance Manager	Severity:	S1 (Critical)
Version:	5.0.32	OS:	Linux (2.6.9-34)
Assigned to:		CPU Architecture:	Any
Tags:	fails, Instance, shutdown

Description:
mysql  Ver 14.12 Distrib 5.0.32, for pc-linux-gnu (i686) using readline 5.0.
Redhat Linux 2.6.9-34.ELsmp #1 SMP Fri Feb 24 16:54:53 EST 2006 i686 i686 i386 GNU/Linux.

Just to re-iterate. We do not have an issue with the instance exiting unexpectedly. 
Our problem is that sporadically the instance manager fails to stop an instance when requested, but does not show this error and falsely reports a successful shutdown. 

Is there an fix available for this in later versions of mysql?
Answer Urgently requested. 

How to repeat:
Unfortunately there is not one specific situation in which it fails, so I do not know how to reproduce this.

Thank you for the bug report. The Instance Manager will be discontinued.

Miguel,

You write the instance manager will be discontinued.
In which release of MYSQL?

Thanks in advance,
Hans Habers

I would like to know in which release Instance Manager will be discontinued.

Do MySQL therefore recommend not using the Instance Manager to stop and start instances ?

If so, what is then recommended procedure ?

Thank you for the feedback.

Currently you can use mysqld_multi which is old and tested by time tool.

Btw how do you stop an instance? Do you issue SHOW INSTANCES command after doing this?

No feedback was provided for this bug for over a month, so it is
being suspended automatically. If you are able to provide the
information that was originally requested, please do so and change
the status of the bug back to "Open".

I see this as well.  I am running version 5.0.54.  I understand that this tool is going away; however, it's what we have for now.  I'm using it to run a replicated db where I have a script that uses the mysqlmanager to stop and restart the replication in order to make a snap of the Innodb files for backup.

Here's a failure example:

mysql> show instances;
+---------------+--------+
| instance_name | status |
+---------------+--------+
| mysqld1       | online | 
| mysqld        | online | 
+---------------+--------+
2 rows in set (0.00 sec)

mysql> stop instance mysqld1;
Query OK, 0 rows affected (0.00 sec)

mysql> show instances;
+---------------+--------+
| instance_name | status |
+---------------+--------+
| mysqld1       | online | 
| mysqld        | online | 
+---------------+--------+
2 rows in set (0.00 sec)

One hint that I find is that there was an error when mysqlmanager initially started up this instance.  I did not observe the error at the time that it occurred, but I can tell you that it appears as if mysqlmanager sometimes starts up two instances of the same name and that one dies after a short while and the other lives on (but is not known to the IM).  

I see this in the mysql error log:

081217 16:51:14  InnoDB: Started; log sequence number 23 543078864
081217 16:51:15 [Note] /usr/sbin/mysqld: ready for connections.
Version: '5.0.54-log'  socket: '/var/run/mysqld/mysqld-bkup.sock'  port: 3316  Gentoo Linux mysql-5.0.54
081217 16:51:15 [Note] Slave SQL thread initialized, starting replication in log 'cl53-repl-log.000086' at position 773519783, relay log '/data/mysql-bkup_log/relay/cl49-bkup-relay-log.001150' position: 406580812
081217 16:51:15 [Note] Slave I/O thread: connected to master 'repl@cl53:3306',  replication started in log 'cl53-repl-log.000086' at position 773519783
InnoDB: Unable to lock ./ibdata1, error: 11
InnoDB: Check that you do not already have another mysqld process
InnoDB: using the same InnoDB data or log files.
081217 16:51:14  InnoDB: Retrying to lock the first data file
InnoDB: Unable to lock ./ibdata1, error: 11
InnoDB: Check that you do not already have another mysqld process
InnoDB: using the same InnoDB data or log files.

<these few lines repeat a number of times>

InnoDB: Check that you do not already have another mysqld process
InnoDB: using the same InnoDB data or log files.
081217 16:52:54  InnoDB: Unable to open the first data file
InnoDB: Error in opening ./ibdata1
081217 16:52:54  InnoDB: Operating system error number 11 in a file operation.
InnoDB: Error number 11 means 'Resource temporarily unavailable'.
InnoDB: Some operating system error numbers are described at
InnoDB: http://dev.mysql.com/doc/refman/5.0/en/operating-system-error-codes.html
InnoDB: Could not open or create data files.
InnoDB: If you tried to add new data files, and it failed here,
InnoDB: you should now edit innodb_data_file_path in my.cnf back
InnoDB: to what it was, and remove the new ibdata files InnoDB created
InnoDB: in this failed attempt. InnoDB only wrote those files full of
InnoDB: zeros, but did not yet use them in any way. But be careful: do not
InnoDB: remove old data files which contain your precious data!
081217 16:52:54 [ERROR] Can't start server: Bind on TCP/IP port: Address already in use
081217 16:52:54 [ERROR] Do you already have another mysqld server running on port: 3316 ?
081217 16:52:54 [ERROR] Aborting

081217 16:52:54 [Note] /usr/sbin/mysqld: Shutdown complete

I can report that this also happens in version 5.0.70