MySQL Bugs: #55015: MySQL server is not restarted properly after RPM upgrade

Bug #55015	MySQL server is not restarted properly after RPM upgrade
Submitted:	5 Jul 2010 22:45	Modified:	4 Sep 2010 15:30
Reporter:	Elena Stepanova	Email Updates:
Status:	Closed	Impact on me:	None
Category:	MySQL Server: Installing	Severity:	S3 (Non-critical)
Version:	5.5.5-m3	OS:	Linux
Assigned to:	Joerg Bruehe	CPU Architecture:	Any
Tags:	regression

Description:
According to bug#27072, if by the time when RPM upgrade starts MySQL server is running, it should be restarted after upgrade is finished.

It does not work exactly this way now.

When RPM upgrade is started, the old MySQL server is shut down, upgrade is performed and the new MySQL server is launched. However, as soon as the upgrade process ends, MySQL server is shut down again.

There are no errors in MySQL err log, server goes through normal shutdown:

--- tail of server err log ---

100706  0:30:41 [Note] /usr/sbin/mysqld: Normal shutdown

100706  0:30:41 [Note] Event Scheduler: Purging the queue. 0 events
100706  0:30:41  InnoDB: Starting shutdown...
100706  0:30:42  InnoDB: Shutdown completed; log sequence number 44244
100706  0:30:42 [Note] /usr/sbin/mysqld: Shutdown complete

100706 00:30:42 mysqld_safe mysqld from pid file /var/lib/mysql/ff.pid ended
100706 00:30:58 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
100706  0:30:58 [Note] Buffered information: Performance schema disabled (reason: start parameters).

100706  0:30:58 [Note] Plugin 'FEDERATED' is disabled.
InnoDB: The InnoDB memory heap is disabled
InnoDB: Mutexes and rw_locks use GCC atomic builtins
InnoDB: Compressed tables use zlib 1.2.3
100706  0:30:58  InnoDB: Using Linux native AIO
100706  0:30:58  InnoDB: highest supported file format is Barracuda.
InnoDB: 127 rollback segment(s) active.
100706  0:30:58 InnoDB 1.1.1 started; log sequence number 44244
100706  0:30:58 [Note] Event Scheduler: Loaded 0 events
100706  0:30:58 [Note] /usr/sbin/mysqld: ready for connections.
Version: '5.5.5-m3'  socket: '/var/lib/mysql/mysql.sock'  port: 3306  MySQL Community Server (GPL)
100706  0:31:03 [Note] /usr/sbin/mysqld: Normal shutdown

100706  0:31:03 [Note] Event Scheduler: Purging the queue. 0 events
100706  0:31:03  InnoDB: Starting shutdown...
100706  0:31:04  InnoDB: Shutdown completed; log sequence number 1589350
100706  0:31:04 [Note] /usr/sbin/mysqld: Shutdown complete

100706 00:31:04 mysqld_safe mysqld from pid file /var/lib/mysql/ff.pid ended

--- end of server err log ---

RPM_UPGRADE_HISTORY also does not report any problems:

--- Full RPM_UPGRADE_HISTORY ---

MySQL RPM upgrade to version 5.5.5-m3-1.sles10
'pre' step running at Tue Jul  6 00:30:41 CEST 2010

ERR file(s):
-rw-rw---- 1 mysql users 1431 Jul  6 00:29 /var/lib/mysql/ff.err

Latest 'Version' line in latest file:
Version: '5.5.3-m3-community'  socket: '/var/lib/mysql/mysql.sock'  port: 3306  MySQL Community Server (GPL)

PID file:
-rw-rw---- 1 mysql mysql 6 Jul  6 00:29 /var/lib/mysql/ff.pid
31926

Server process:
UID        PID  PPID  C STIME TTY          TIME CMD
mysql    31926 31847  0 00:29 pts/6    00:00:00 /usr/sbin/mysqld --basedir=/ --datadir=/var/lib/mysql --plugin-dir=/usr/lib64/mysql/plugin --user=mysql --log-error=/var/lib/mysql/ff.err --pid-file=/var/lib/mysql/ff.pid

SERVER_TO_START=true
MySQL RPM upgrade to version 5.5.5-m3-1.sles10
Upgrade/install finished at Tue Jul  6 00:31:03 CEST 2010

=====
--- End of RPM_UPGRADE_HISTORY ---

Tried SLES 10 PATCH 2 (x86_64) and RHEL 4 (i386).

It does not happen when I upgrade from 5.5.2 to 5.5.3, server keeps running after upgrade.

How to repeat:
rpm -ivh MySQL-server-community-5.5.3_m3-1.sles10.x86_64.rpm
# wait for installation to finish
# check that mysqld is running
rpm -Uvh <5.5.5_m3 community server>
# wait for upgrade to finish
# see that mysqld is not running

I cannot reproduce the problem.

Using a SuSE 11 (virtual) machine with x86_64,
I installed the "generic" RPMs (client, server, shared) of 5.5.3-m3 and upgraded to 5.5.5-m3. The server was running when I started the upgrade, and the new server was running after the upgrade.
Old RPMs:  MySQL-{client,server,shared}-5.5.3_m3-1.glibc23.x86_64.rpm
New RPMs:  MySQL-{client,server,shared}-5.5.5_m3-1.linux2.6.x86_64.rpm

It worked equally well with those for SLES 11
   MySQL-{client,server,shared}-community-5.5.3_m3-1.sles11.x86_64.rpm
   MySQL-{client,server,shared}5.5.5_m3-1.sles11.x86_64.rpm
    
Same positive result with those for SLES 10:
   MySQL-{client,server,shared}-community-5.5.3_m3-1.sles10.x86_64.rpm
   MySQL-{client,server,shared}5.5.5_m3-1.sles10.x86_64.rpm

In all tests, "ps -afe | fgrep mysql" showed the server to be running well after the "rpm -U" finished.

My database was empty, I did not load any data before testing.
If your environment or result are different, please provide info.

Sorry - "need feedback" is more appropriate.

Hi Joerg,

Yes, both my environment and result in the initial test were different.
Result was as stated before -- the process would not run after upgrade.
Environment/test differences:
1) I did not install *shared* packages;
2) I was not using VM;
3) I was running the test on SLES 10.2, not SLES 11;
4) I tried sles10 packages, but not generic Linux.

------
Basic observations

1) The difference with shared packages is not directly related to the problem, so I'll get back to it later.

2) To eliminate the VM factor, I've installed three new clean virtual (32-bit) boxes: SLES 10.3, SLES 11 and SLES 11.1, and repeated the experiment using them. SLES 10.3 under VM behaved the same way as the real SLES 10.2 box I was using before, so the VM part is irrelevant; and SLES 11 and SLES 11.1 gave identical (to each other) results, so SPs are irrelevant also; further I will refer to results simply as SLES 10 vs SLES 11.

3) SLES 10 and SLES 11 (with RPM 4.4.2 vs 4.4.2.3, correspondingly) produce essentially different results for this sequence:

rpm -ivh MySQL-server-community-5.5.3_m3-1.sles10.i586.rpm
rpm -Uvh [--force] MySQL-server-5.5.5_m3-1.sles10.i586.rpm

SLES 11 does not allow upgrade in normal mode (without 'force'). It complains about conflicts with the previously installed 5.5.3 package, e.g.

...
file /usr/bin/myisampack from install of MySQL-server-5.5.5_m3-1.sles10.i586 conflicts with file from package MySQL-server-community-5.5.3_m3-1.sles10.i586
...

etc., and aborts. With the --force option, it installs the 5.5.5 package, but does not erase 5.5.3, so both packages are seen via RPM query. So essentially, it performs install, not upgrade; and as you said, mysqld process is up and running after RPM has done the work.

SLES 10 allows the second command without 'force', and performs the upgrade. 5.5.5 package is installed, and 5.5.3 is erased (not shown by RPM query afterwards). However, mysqld process is not up after upgrade. Also, it is not started automatically after reboot. It can still be run manually by '/etc/init.d/mysql start'.

4) Generic packages work OK -- upgrade is performed, server is up.

------
Debug

From RPM debug output, it looks like due to renaming (?) done for non-generic packages, RPM is asked to erase 5.5.3; and it attempts to do so on SLES 10, at least it recognizes the request to obsolete/erase the package:

...
D:   Obsoletes: MySQL           erases MySQL-server-community-5.5.3_m3-1.sles10.i586
D:      added binary package [0]
D: found 0 source and 1 binary packages
D: ========== +++ MySQL-server-5.5.5_m3-1.sles10 i586/linux 0x0
...

while on SLES 11 RPM does not say anything of the kind, as if it completely ignores the request:

D: opening  db environment /var/lib/rpm/Packages create:cdb:mpool:private
D: opening  db index       /var/lib/rpm/Packages rdonly mode=0x0
D: locked   db index       /var/lib/rpm/Packages
D: opening  db index       /var/lib/rpm/Name rdonly:nofsync mode=0x0
D:      added binary package [0]
D: found 0 source and 1 binary packages
D: ========== +++ MySQL-server-5.5.5_m3-1.sles10 i586/linux 0x0
D: opening  db index       /var/lib/rpm/Depends create:nofsync mode=0x0
D: opening  db index       /var/lib/rpm/Providename rdonly:nofsync mode=0x0
D: opening  db index       /var/lib/rpm/Pubkeys rdonly:nofsync mode=0x0
D:  read h#       4 Header sanity check: OK

Further during upgrade, after all the new package has been installed and server started, RPM on SLES 10 runs preun for 5.5.3, and does so as if it wants to totally eliminate MySQL server:

...
Starting MySQL...done
+ echo 'Giving mysqld 2 seconds to start'
Giving mysqld 2 seconds to start
+ sleep 2
+ sleep 2
++ date
+ echo 'Upgrade/install finished at Sun Aug  8 18:32:12 CEST 2010'
+ echo
+ echo =====
+ STATUS_HISTORY=/var/lib/mysql/RPM_UPGRADE_HISTORY
+ cat /var/lib/mysql/RPM_UPGRADE_MARKER
+ rm /var/lib/mysql/RPM_UPGRADE_MARKER
D:   install: waitpid(17123) rc 17123 status 0 secs 5.832
D: opening  db index       /var/lib/rpm/Triggername create:nofsync mode=0x42
D: ========== --- MySQL-server-community-5.5.3_m3-1.sles10 x86_64-linux 0x0
D:     erase: MySQL-server-community-5.5.3_m3-1.sles10 has 175 files, test = 0
D:     erase: %preun(MySQL-server-community-5.5.3_m3-1.sles10.x86_64) asynchronous scriptlet start
D:     erase: %preun(MySQL-server-community-5.5.3_m3-1.sles10.x86_64)   execv(/bin/sh) pid 17278
+ '[' 0 = 0 ']'
+ '[' -x /etc/init.d/mysql ']'
+ /etc/init.d/mysql stop
+ '[' -x /sbin/chkconfig ']'
+ /sbin/chkconfig --del mysql
mysql                     0:off  1:off  2:off  3:off  4:off  5:off  6:off
D:     erase: waitpid(17278) rc 17278 status 0 secs 1.086
...

So, the process gets stopped again right after it started (thus there is no mysqld running after upgrade), and the service is also disabled (no automatic startup after reboot). 

Nothing like this happens for generic packages, apparently because there was no renaming for them, and the regular upgrade is performed.

------
Additional note

Back to *shared* packages, they are not upgraded properly even on SLES 10 -- RPM also complains about conflicts, does not show any signs of 'obsolete' requests, and under 'force' flag does not remove 5.5.3 package after installing 5.5.5.

------
Conclusions

I have no knowledge whatsoever about RPM magic, so I could only make guesses, sorry if they do not make much sense.
Based on everything above, I would think that

* RPMs on SLES 10 and SLES 11 behave differently in regard to our request to erase previous packages -- RPM on SLES 10 recognizes it, and on SLES 11 does not;
* preun job for the server does not expect the 'obsolete' flow and behaves as if it were un-installation.
* shared packages do not provide RPM with enough information to recognize 'obsolete' request.

Please let me know if you want me to provide any other information.

I admit that in my tests I had missed a point:
When upgrading from 5.5.3 to 5.5.5, the old version did not get removed.
I had no explanation for this, especially as they both "provide" the same functions:
   msqlormysql
   mysql-server
   mysql
   MySQL
   MySQL-server
(plus stuff about plugins which should hopefully not matter to much).

However, I now learned that with a change of the package (file) name the new one should explicitly "obsolete" the old one, and I trust this is missing.
The "-community" part must have fallen victim to some spec file cleanup / alignment.

I will see to change the spec file and re-build 5.5.5, not changing anything but the spec file (adding "obsoletes") and setting the "release" to 2.
I hope to have first results for testing available tonight or tomorrow.

I successfully built new RPMs 5.5.5_m3-2 "sles11" (x86_64 only) with the "Obsoletes" specification and tried them:
They can be used to upgrade 5.5.3_m3 "community-sles11" without claiming a conflict and requiring "--force".

However, after such an upgrade the server is not running.

I noticed the same line as reported:
   mysql   0:off  1:off  2:off  3:off  4:off  5:off  6:off
and I suspect this is related to the bug, but I cannot yet tell what causes it.

The start/stop script "/etc/init.d/mysql" looks correct (specifies run levels 2, 3, 4, and 5), so I will have to inquire deeper.

A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/115534

3101 Joerg Bruehe	2010-08-11
      Fix Bug#55015
        "MySQL server is not restarted properly after RPM upgrade"
      
      The problem is that with the general spec file cleanup and
      alignment we also did a name change, dropping the "-community"
      part from the package file name.
      
      As a result of this, RPM (some versions of it) will report
      file conflicts, because it considers this name difference
      to imply different packages.
      To avoid this, the spec file explicitly "obsoletes" the old
      packages (with "-community" in the file name).
      
      Now, RPM will first install these packages and the remove the
      old ones, and part of that removal is running the "%preun"
      section which stops the server and uninstalls the service
      (removes the symlinks to "/etc/init.d/mysql" from the run
      level directories).
      This stop/uninstall will affect the new server!
      
      The fix is to define a "%triggerpostun" in this spec file
      which will watch for removal of the "-community" server.
      If this is done (as part of this install/upgrade), the
      trigger code will re-install the service and restart the
      server process.
      
      In addition, the "sleep" calls after starting the server
      have been cleaned up: Rather than doing 2* "sleep 2",
      it is now 1 "sleep 5".

Lenz approved via IRC, after csearching the Web and coming to the same conclusion as I did:
This problem during an "obsoletes"-uninstall is a design problem of RPM,
we cannot avoid using a trigger to overcome the "%preun" actions.

A test build on all RPM platforms is running.

Test build looked good on all RPM platforms, passed verification.
Fix is pushed to "5.5-bugfixing" (currently at version 5.5.6-m3).

Pushed into mysql-5.5 5.5.6-m3 (revid:alik@ibmvm-20100825092002-2yvkb3iwu43ycpnm) (version source revid:alik@ibmvm-20100825092002-2yvkb3iwu43ycpnm) (merge vers: 5.5.6-m3) (pib:20)

A note has been added to the 5.5.6 changelog: 

        When upgrading an existing install with an RPM on Linux, the                                                                                       
        MySQL server might not have been restarted properly. This was                                                                                      
        due to a naming conflict when upgrading from                                                                                                       
        a <literal>community</literal> named RPM. Previous                                                                                                 
        installations are now correctly removed, and the MySQL init                                                                                        
        script are recreated and then start the MySQL server as                                                                                            
        normal.

Pushed into mysql-trunk 5.6.1-m4 (revid:alik@sun.com-20100830082732-n2eyijnv86exc5ci) (version source revid:alik@sun.com-20100830082732-n2eyijnv86exc5ci) (merge vers: 5.6.1-m4) (pib:21)

Pushed into mysql-next-mr (revid:alik@sun.com-20100830082745-n6sh01wlwh3itasv) (version source revid:alik@sun.com-20100830082745-n6sh01wlwh3itasv) (pib:21)

Added to the 5.6.1 changelog.