MySQL Bugs: #96696: [MY-013379] [Server] Server upgrade started with version 80017

Bug #96696	[MY-013379] [Server] Server upgrade started with version 80017
Submitted:	29 Aug 2019 12:08	Modified:	2 Oct 2019 13:18
Reporter:	Sergej Begizov	Email Updates:
Status:	No Feedback	Impact on me:	None
Category:	MySQL Server	Severity:	S1 (Critical)
Version:	80015	OS:	CentOS
Assigned to:	MySQL Verification Team	CPU Architecture:	Any
Tags:	MY-013379

Description:
Unable to start mysql on centos

[ERROR] [MY-013379] [Server] Server upgrade started with version 80017, but server upgrade of version 80015 is still pending.
[ERROR] [MY-010020] [Server] Data Dictionary initialization failed.
[ERROR] [MY-010119] [Server] Aborting

Was working until ~2019-07-06 and now i cannot start MySQL.
Please let me know if any additional info is necessary.
Searched for this error on the web, but did not find a single reference

How to repeat:
i havo no idea how to repeat this, nor how to fix it

Fixed it by rolling back latest update

Hi,

Can you please give more details.

I upgraded without any issues, on CentOS 7, 8.0.15 to 8.0.16 to 8.0.17

please update 
 - What version of CentOS
 - How did you do update/upgrade, did you manually yum update or this is from a cron or you were doing something else
 - What repo you have configured for mysql

thanks

I'm really new to this, so please help me if i'm stuck :)
Centos version: centos-release-7-6.1810.2.el7.centos.x86_64

about updates.
I've installed mysql in ~2019-02 using some manual on the web. so i did not specially configure any cron jobs for the Mysql, nor did i do a manual update.

in ~2019-04 i've noticed that something has changed, since i begun receiving errors about DateTime format, since i did not know what happened i just added UTC fo my connection string.

in July i was on a vacation, so nobody saw that mysql is not working.
on monday i tryed to connect to my db and it was unavailable.
i connected to my VPS and mysqld service was down (systemctl status mysqld).
then i tryed to start it manually (systemctl start mysqld) reveived an error.
check the logs and saw an error posted in my question.
Since i'm a windows guy, i thougnt that restarting OS would allow updates to be installed (silly me), nothen happened. error log stated that new update is not able to be installed since previous update is pending.

then i posted this bug report.

then i used yum history list all to get a list of all updates.
then i found the latest mysql update using yum list info ? (manually all id one-by-one from the top)
then i rolled back using yum history undo 76 (id of the update)

then i was able to start mysqld service

Hi,

Well, you need a real system admin to setup and maintain that server, getting help setting it up trough bugs system is not really going to work, better place would be https://forums.mysql.com/

Now, I have no clue what that web manual made you do in 2009 but check following

1. where are you getting your mysql binaries. To check that go to  /etc/yum.repos.d/ and check if you have mysql-community.repo there

then look at that file and see [mysql80-community] section and there should be something like :

baseurl=http://repo.mysql.com/yum/mysql-8.0-community/el/7/$basearch/
enabled=1

check if that's so, or if you have something else configured there.

now in order to see why the last update failed you need to find mysql log, check for log-error line in /etc/my.cnf and check for /var/log/mysqld.log file and check if in /var/log/messages you also have mysqld logs.

then there you need to find the failed start after that failed upgrade and get the log entries to see why it did not start / what failed.

Errors in the error log are pretty self explanatory and you will most probably be able to solve a problem yourself, but feel free to paste the errors here so we can see if maybe there's actually really a bug there :)

all best

ofcourse i need a real admin, but currentlly i'm a one man army :)
i would not post this as a bug report if i would find any mension of this error code on the web anywere :)

1. i have this in mysql-community.repo
[mysql80-community]
name=MySQL 8.0 Community Server
baseurl=http://repo.mysql.com/yum/mysql-8.0-community/el/7/$basearch/
enabled=1
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-mysql

2. mysqlg.log part where i see any errors 
2019-07-22T00:55:36.662968Z 0 [System] [MY-010910] [Server] /usr/sbin/mysqld: Shutdown complete (mysqld 8.0.16)  MySQL Community Server - GPL.
2019-07-22T00:55:38.134518Z 0 [System] [MY-010116] [Server] /usr/sbin/mysqld (mysqld 8.0.17) starting as process 22178
2019-07-22T00:55:39.599540Z 1 [ERROR] [MY-013379] [Server] Server upgrade started with version 80017, but server upgrade of version 80015 is still pending.
2019-07-22T00:55:39.600054Z 0 [ERROR] [MY-010020] [Server] Data Dictionary initialization failed.
2019-07-22T00:55:39.602656Z 0 [ERROR] [MY-010119] [Server] Aborting
2019-07-22T00:55:40.127145Z 0 [System] [MY-010910] [Server] /usr/sbin/mysqld: Shutdown complete (mysqld 8.0.17)  MySQL Community Server - GPL.
2019-08-29T09:51:53.041211Z 0 [System] [MY-010116] [Server] /usr/sbin/mysqld (mysqld 8.0.17) starting as process 17008
2019-08-29T09:51:53.442712Z 1 [ERROR] [MY-013379] [Server] Server upgrade started with version 80017, but server upgrade of version 80015 is still pending.
2019-08-29T09:51:53.442993Z 0 [ERROR] [MY-010020] [Server] Data Dictionary initialization failed.
2019-08-29T09:51:53.443308Z 0 [ERROR] [MY-010119] [Server] Aborting
2019-08-29T09:51:53.965468Z 0 [System] [MY-010910] [Server] /usr/sbin/mysqld: Shutdown complete (mysqld 8.0.17)  MySQL Community Server - GPL.
2019-08-29T09:52:11.971055Z 0 [System] [MY-010116] [Server] /usr/sbin/mysqld (mysqld 8.0.17) starting as process 17063
2019-08-29T09:52:12.343154Z 1 [ERROR] [MY-013379] [Server] Server upgrade started with version 80017, but server upgrade of version 80015 is still pending.
2019-08-29T09:52:12.343559Z 0 [ERROR] [MY-010020] [Server] Data Dictionary initialization failed.
2019-08-29T09:52:12.344093Z 0 [ERROR] [MY-010119] [Server] Aborting
2019-08-29T09:52:12.865265Z 0 [System] [MY-010910] [Server] /usr/sbin/mysqld: Shutdown complete (mysqld 8.0.17)  MySQL Community Server - GPL.
2019-08-29T10:01:39.482706Z 0 [System] [MY-010116] [Server] /usr/sbin/mysqld (mysqld 8.0.17) starting as process 1170
2019-08-29T10:01:45.486297Z 1 [ERROR] [MY-013379] [Server] Server upgrade started with version 80017, but server upgrade of version 80015 is still pending.
2019-08-29T10:01:45.486602Z 0 [ERROR] [MY-010020] [Server] Data Dictionary initialization failed.
2019-08-29T10:01:45.501760Z 0 [ERROR] [MY-010119] [Server] Aborting
2019-08-29T10:01:45.879850Z 0 [System] [MY-010910] [Server] /usr/sbin/mysqld: Shutdown complete (mysqld 8.0.17)  MySQL Community Server - GPL.
2019-08-29T13:13:37.931026Z 0 [System] [MY-010116] [Server] /usr/sbin/mysqld (mysqld 8.0.17) starting as process 12516
2019-08-29T13:13:38.404885Z 1 [ERROR] [MY-013379] [Server] Server upgrade started with version 80017, but server upgrade of version 80015 is still pending.
2019-08-29T13:13:38.405290Z 0 [ERROR] [MY-010020] [Server] Data Dictionary initialization failed.
2019-08-29T13:13:38.405757Z 0 [ERROR] [MY-010119] [Server] Aborting
2019-08-29T13:13:38.923939Z 0 [System] [MY-010910] [Server] /usr/sbin/mysqld: Shutdown complete (mysqld 8.0.17)  MySQL Community Server - GPL.
2019-08-29T13:14:58.942920Z 0 [System] [MY-010116] [Server] /usr/sbin/mysqld (mysqld 8.0.17) starting as process 12593
2019-08-29T13:14:59.381189Z 1 [ERROR] [MY-013379] [Server] Server upgrade started with version 80017, but server upgrade of version 80015 is still pending.
2019-08-29T13:14:59.381445Z 0 [ERROR] [MY-010020] [Server] Data Dictionary initialization failed.
2019-08-29T13:14:59.381643Z 0 [ERROR] [MY-010119] [Server] Aborting
2019-08-29T13:14:59.896257Z 0 [System] [MY-010910] [Server] /usr/sbin/mysqld: Shutdown complete (mysqld 8.0.17)  MySQL Community Server - GPL.
2019-08-29T13:17:39.316722Z 0 [System] [MY-010116] [Server] /usr/sbin/mysqld (mysqld 8.0.17) starting as process 12663
2019-08-29T13:17:39.741459Z 1 [ERROR] [MY-013379] [Server] Server upgrade started with version 80017, but server upgrade of version 80015 is still pending.
2019-08-29T13:17:39.741917Z 0 [ERROR] [MY-010020] [Server] Data Dictionary initialization failed.
2019-08-29T13:17:39.742442Z 0 [ERROR] [MY-010119] [Server] Aborting
2019-08-29T13:17:40.263283Z 0 [System] [MY-010910] [Server] /usr/sbin/mysqld: Shutdown complete (mysqld 8.0.17)  MySQL Community Server - GPL.
2019-08-29T13:19:05.925999Z 0 [System] [MY-010116] [Server] /usr/sbin/mysqld (mysqld 8.0.17) starting as process 12747
2019-08-29T13:19:06.364658Z 1 [ERROR] [MY-013379] [Server] Server upgrade started with version 80017, but server upgrade of version 80015 is still pending.
2019-08-29T13:19:06.365081Z 0 [ERROR] [MY-010020] [Server] Data Dictionary initialization failed.
2019-08-29T13:19:06.365707Z 0 [ERROR] [MY-010119] [Server] Aborting
2019-08-29T13:19:06.886433Z 0 [System] [MY-010910] [Server] /usr/sbin/mysqld: Shutdown complete (mysqld 8.0.17)  MySQL Community Server - GPL.
2019-08-29T13:45:16.454289Z 0 [System] [MY-010116] [Server] /usr/sbin/mysqld (mysqld 8.0.16) starting as process 13085
2019-08-29T13:45:17.693721Z 0 [Warning] [MY-013378] [Server] Server upgrade is required, but skipped by command line option '--upgrade=MINIMAL'.
2019-08-29T13:45:17.837027Z 0 [Warning] [MY-010068] [Server] CA certificate ca.pem is self signed.
2019-08-29T13:45:17.928194Z 0 [System] [MY-010931] [Server] /usr/sbin/mysqld: ready for connections. Version: '8.0.16'  socket: '/var/lib/mysql/mysql.sock'  port: 3306  MySQL Community Server - GPL.
2019-08-29T13:45:17.931855Z 0 [System] [MY-011323] [Server] X Plugin ready for connections. Socket: '/var/run/mysqld/mysqlx.sock' bind-address: '::' port: 33060
2019-08-29T14:40:11.650536Z 0 [System] [MY-010910] [Server] /usr/sbin/mysqld: Shutdown complete (mysqld 8.0.16)  MySQL Community Server - GPL.
2019-08-29T14:40:40.520524Z 0 [System] [MY-010116] [Server] /usr/sbin/mysqld (mysqld 8.0.16) starting as process 1104
2019-08-29T14:40:47.203994Z 0 [Warning] [MY-013378] [Server] Server upgrade is required, but skipped by command line option '--upgrade=MINIMAL'.
2019-08-29T14:40:47.429928Z 0 [Warning] [MY-010068] [Server] CA certificate ca.pem is self signed.
2019-08-29T14:40:47.756633Z 0 [System] [MY-010931] [Server] /usr/sbin/mysqld: ready for connections. Version: '8.0.16'  socket: '/var/lib/mysql/mysql.sock'  port: 3306  MySQL Community Server - GPL.
2019-08-29T14:40:47.921810Z 0 [System] [MY-011323] [Server] X Plugin ready for connections. Socket: '/var/run/mysqld/mysqlx.sock' bind-address: '::' port: 33060
3. messages do not have anything about mysqld

Please close this bug report if this is actually not a bug, but my crooked hands :)

Hi,

I think it's not a bug but I'd like to get that server updated just to be sure there's not a bug hidden somewhere.

The repo data is ok, you are pulling proper binaries.

Are you running "yum update" yourself or yum is updateing from a cron.

You can check 
/etc/crontab
/etc/cron.d/*
/etc/cron.hourly/*
/etc/cron.daily/*
/etc/cron.weekly/*
/etc/cron.monthly/*
/var/spool/cron/*/*

see if you have yum update or something similar set to run automatically

If I understood you correctly so far you have 8.0.15 running properly now.

running "mysql_upgrade" on it's own it will check current 8.0.15 and upgrade stuff if requred (system tables, data ..) 

Then, you can try running "yum update mysql mysql-community-client mysql-community-common mysql-community-server"  manually, follow the process and see if it finishes as expected.

yes i have 
#!/bin/bash
yum update -y 
yum clean all
rm -rf /var/cache/yum
in crod.daily

mysql version actually now is 
# mysql --version
mysql  Ver 8.0.17 for Linux on x86_64 (MySQL Community Server - GPL)

i did not run update manually, but it is updated from yum (probobly default centos setting, did not set myself)

Hi,

nope, that script is not "default centos", someone put that there :)

so if mysql is 8.0.17 it updated. run manually "mysql_upgrade" to check if upgrade finished properly (and to finish it if needed)

i've read that mysql_upgrade is depricaeted

how should i update mysql manually from server executable?

mysqld --upgrade=FORCE? 
or 
mysqld --upgrade=MINIMAL? 

# mysql_upgrade
The mysql_upgrade client is now deprecated. The actions executed by the upgrade client are now done by the server.
To upgrade, please start the new MySQL binary with the older data directory. Repairing user tables is done automatically. Restart is not required after upgrade.
The upgrade process automatically starts on running a new MySQL binary with an older data directory. To avoid accidental upgrades, please use the --upgrade=NONE option with the MySQL binary. The option --upgrade=FORCE is also provided to run the server upgrade sequence on demand.
It may be possible that the server upgrade fails due to a number of reasons. In that case, the upgrade sequence will run again during the next MySQL server start. If the server upgrade fails repeatedly, the server can be started with the --upgrade=MINIMAL option to start the server without executing the upgrade sequence, thus allowing users to manually rectify the problem.

Hi,

It is deprecated but it still works :D until it is removed from the mysql package (it is still there in 8.0.17 iirc). Benefit from mysql_upgrade is that you don't need to shutdown your running server.

You can run mysqld --upgrade (no need to add any parameters, default is --upgrade=auto) without parameters and if that does not solve the issue then you can --upgrade=FORCE

all best

[root@b5ecc ~]# mysqld --upgrade
[root@b5ecc ~]# mysqld --version
/usr/sbin/mysqld  Ver 8.0.17 for Linux on x86_64 (MySQL Community Server - GPL)
[root@b5ecc ~]#

so i guess i'm up-to-date?

Check the log, but it seems ok.

nope, not ok
just nothing happened

2019-09-02T11:07:19.125917Z 0 [ERROR] [MY-000071] [Server] /usr/sbin/mysqld: option '--upgrade' requires an argument.
2019-09-02T11:07:19.126386Z 0 [ERROR] [MY-010119] [Server] Aborting
2019-09-02T11:07:19.126558Z 0 [Note] [MY-010120] [Server] Binlog end

now i can't start mysql again, with same error

2019-09-02T12:10:42.368215Z 0 [System] [MY-010116] [Server] /usr/sbin/mysqld (mysqld 8.0.17) starting as process 23259
2019-09-02T12:10:42.851944Z 1 [ERROR] [MY-013379] [Server] Server upgrade started with version 80017, but server upgrade of version 80015 is still pending.
2019-09-02T12:10:42.852315Z 0 [ERROR] [MY-010020] [Server] Data Dictionary initialization failed.
2019-09-02T12:10:42.852631Z 0 [ERROR] [MY-010119] [Server] Aborting
2019-09-02T12:10:43.375498Z 0 [System] [MY-010910] [Server] /usr/sbin/mysqld: Shutdown complete (mysqld 8.0.17)  MySQL Community Server - GPL.

downgraded to 8.0.16 and was able to start server
i suppose tomorrow cron will update it to 8.0.17 again...

Hi,

So, as I wrote, if --upgrade without parameters don't solve it (should assume AUTO but..) use --upgrade=FORCE

Ok,
i ran this
# mysqld --upgrade=FORCE -u root

got this in log's

2019-09-02T12:37:46.647210Z 0 [System] [MY-010116] [Server] /usr/sbin/mysqld (mysqld 8.0.16) starting as process 23646
2019-09-02T12:37:46.687450Z 1 [ERROR] [MY-012574] [InnoDB] Unable to lock ./ibdata1 error: 11
#skiped multiple lock error, because whole log was too long
2019-09-02T12:39:25.720628Z 1 [ERROR] [MY-012574] [InnoDB] Unable to lock ./ibdata1 error: 11
2019-09-02T12:39:26.720916Z 1 [ERROR] [MY-012574] [InnoDB] Unable to lock ./ibdata1 error: 11
2019-09-02T12:39:26.722760Z 1 [ERROR] [MY-012592] [InnoDB] Operating system error number 11 in a file operation.
2019-09-02T12:39:26.722814Z 1 [ERROR] [MY-012596] [InnoDB] Error number 11 means 'Resource temporarily unavailable'
2019-09-02T12:39:26.722853Z 1 [ERROR] [MY-012215] [InnoDB] Cannot open datafile './ibdata1'
2019-09-02T12:39:26.722942Z 1 [ERROR] [MY-012959] [InnoDB] Could not open or create the system tablespace. If you tried to add new data files to the system tablespace, and it failed here, you should now edit innodb_data_file_path in my.cnf back to what it was, and remove the new ibdata files InnoDB created in this failed attempt. InnoDB only wrote those files full of zeros, but did not yet use them in any way. But be careful: do not remove old data files which contain your precious data!
2019-09-02T12:39:26.722976Z 1 [ERROR] [MY-012930] [InnoDB] Plugin initialization aborted with error Cannot open a file.
2019-09-02T12:39:27.323854Z 1 [ERROR] [MY-010334] [Server] Failed to initialize DD Storage Engine
2019-09-02T12:39:27.324247Z 0 [ERROR] [MY-010020] [Server] Data Dictionary initialization failed.
2019-09-02T12:39:27.325993Z 0 [ERROR] [MY-010119] [Server] Aborting
2019-09-02T12:39:27.328247Z 0 [System] [MY-010910] [Server] /usr/sbin/mysqld: Shutdown complete (mysqld 8.0.16)  MySQL Community Server - GPL.

> 2019-09-02T12:39:25.720628Z 1 [ERROR] [MY-012574] [InnoDB] Unable to lock ./ibdata1 error: 11

because your mysqld is already running, you need to shut mysql down before running mysqld

No feedback was provided for this bug for over a month, so it is
being suspended automatically. If you are able to provide the
information that was originally requested, please do so and change
the status of the bug back to "Open".