MySQL Bugs: #102297: Error My-013178 after upgrading mysqld 8.0.23

Bug #102297	Error My-013178 after upgrading mysqld 8.0.23
Submitted:	19 Jan 2021 13:33	Modified:	2 Feb 2021 3:49
Reporter:	jeongho lee	Email Updates:
Status:	Unsupported	Impact on me:	None
Category:	MySQL Server	Severity:	S1 (Critical)
Version:	8.0.23	OS:	Ubuntu (5.8.0-38-generic #43-20.04.1)
Assigned to:	MySQL Verification Team	CPU Architecture:	x86

Description:
I am not able to upgrade mysql to 8.0.23 via "apt upgrade"
After the upgrade fail, I am also not able to run mysqld, either.
The log (/var/log/mysql) states the error, my-0130178 saying insert ignore into mysql.db values ... failed with error code = 1136, error message = 'Column count doesn't match value count at row 1'

Is there any way that I can upgrade successfully mysql or restore?

How to repeat:
Hmm

Hi,

Thanks for your report. We need more data.

- from what version you are upgrading from 
- full config (/etc/my.cnf + all other cnf files in .d directory you might have)
- full error log from your attempt to upgrade

Thanks
Bogdan

cnf and error log files

Attachment: mysql_log.tar.gz (application/gzip, text), 1.94 KiB.

I am not sure what version that I upgraded from.
Could you let me know how I can find out?
Thank you.

Hi,

"FROM" should be in the log file, you should see old server being shutdown before upgrade is tried. In the error log you provided there's only a small piece of the log so nothing I can conclude from it.

I tried now upgrade from 5.6 to 8.0 and from 5.7 to 8.0 and from various 8.0 versions to latest 8.0.23 and had no issue with it so without more data from you nothing much I can do.

Thanks
Bogdan

all log files

Attachment: logs.tar.gz (application/gzip, text), 4.52 KiB.

According to your logs, previous version was 8.0.22

e.g.:

> 2021-01-11T23:15:08.412829Z 0 [System] [MY-013172] [Server] Received SHUTDOWN from user <via user signal>. Shutting down mysqld (Version: 8.0.22).

Hi,

I'm having issue reproducing this, on RHEL, CentOS, Oracle Linux, Debian and Ubuntu upgrade works flawlessly. I fear you either had MySQL in some "improper" state before upgrade, some data corruption, or something similar. 

I suggest you contact our support system to help you out of this situation. First thing you want to do is to backup your datadir before you do anything. Then you can try rollback of the upgrade back to 8.0.22 (you can find some data here https://unix.stackexchange.com/questions/79050/can-i-rollback-an-apt-get-upgrade-if-someth... if you don't know how to rollback apt installation) and when you get 8.0.22 up and running do a full check of the system with mysqlcheck https://dev.mysql.com/doc/refman/8.0/en/mysqlcheck.html and a proper backup before you retry the upgrade again.

Reading package lists... Done
Building dependency tree       
Reading state information... Done
mysql-server is already the newest version (8.0.23-1ubuntu20.04).
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
2 not fully installed or removed.
After this operation, 0 B of additional disk space will be used.
Do you want to continue? [Y/n] y
Setting up mysql-community-server (8.0.23-1ubuntu20.04) ...
mysql.conf:23: Line references path below legacy directory /var/run/, updating /var/run/mysqld → /run/mysqld; please update the tmpfiles.d/ drop-in file accordingly.
mysql.conf:23: Line references path below legacy directory /var/run/, updating /var/run/mysqld → /run/mysqld; please update the tmpfiles.d/ drop-in file accordingly.
mysql.conf:23: Line references path below legacy directory /var/run/, updating /var/run/mysqld → /run/mysqld; please update the tmpfiles.d/ drop-in file accordingly.
Job for mysql.service failed because the control process exited with error code.
See "systemctl status mysql.service" and "journalctl -xe" for details.
Job for mysql.service failed because the control process exited with error code.
See "systemctl status mysql.service" and "journalctl -xe" for details.
invoke-rc.d: initscript mysql, action "start" failed.
* mysql.service - MySQL Community Server
     Loaded: loaded (/lib/systemd/system/mysql.service; enabled; vendor preset: enabled)
     Active: failed (Result: exit-code) since Sun 2021-01-24 10:57:10 KST; 3ms ago
       Docs: man:mysqld(8)
             http://dev.mysql.com/doc/refman/en/using-systemd.html
    Process: 3801 ExecStartPre=/usr/share/mysql-8.0/mysql-systemd-start pre (code=exited, status=0/SUCCESS)
    Process: 3840 ExecStart=/usr/sbin/mysqld (code=exited, status=1/FAILURE)
   Main PID: 3840 (code=exited, status=1/FAILURE)
     Status: "Server upgrade complete"
      Error: 2 (No such file or directory)

Jan 24 10:57:04 friend25kr-ubuntu systemd[1]: Starting MySQL Community Server...
Jan 24 10:57:10 friend25kr-ubuntu systemd[1]: mysql.service: Main process exited, code=exited, status=1/FAILURE
Jan 24 10:57:10 friend25kr-ubuntu systemd[1]: mysql.service: Failed with result 'exit-code'.
Jan 24 10:57:10 friend25kr-ubuntu systemd[1]: Failed to start MySQL Community Server.
dpkg: error processing package mysql-community-server (--configure):
 installed mysql-community-server package post-installation script subprocess returned error exit status 1
dpkg: dependency problems prevent configuration of mysql-server:
 mysql-server depends on mysql-community-server (= 8.0.23-1ubuntu20.04); however:
  Package mysql-community-server is not configured yet.

dpkg: error processing package mysql-server (--configure):
 dependency problems - leaving unconfigured
No apport report written because the error message indicates its a followup error from a previous failure
.
 Errors were encountered while processing:
 mysql-community-server
 mysql-server
E: Sub-process /usr/bin/dpkg returned an error code (1)

sudo apt purge mysql-server:amd64=8.0.22-1ubuntu20.04
Reading package lists... Done
Building dependency tree       
Reading state information... Done
E: Version '8.0.22-1ubuntu20.04' for 'mysql-server' was not found

sudo apt install mysql-server:amd64=8.0.22-1ubuntu20.04
Reading package lists... Done
Building dependency tree       
Reading state information... Done
E: Version '8.0.22-1ubuntu20.04' for 'mysql-server' was not found

sudo apt install $(cat ./rollback.txt | tr '\n' ' ')
Reading package lists... Done
Building dependency tree       
Reading state information... Done
E: Version '8.0.22-1ubuntu20.04' for 'mysql-server' was not found
E: Version '8.0.22-1ubuntu20.04' for 'mysql-community-client' was not found
E: Version '8.0.22-1ubuntu20.04' for 'mysql-community-client-core' was not found
E: Version '8.0.22-1ubuntu20.04' for 'libmysqlclient-dev' was not found
E: Version '8.0.22-1ubuntu20.04' for 'libmysqlclient21' was not found
E: Version '8.0.22-1ubuntu20.04' for 'mysql-community-client-plugins' was not found
E: Version '8.0.22-1ubuntu20.04' for 'mysql-community-server-core' was not found
E: Version '8.0.22-1ubuntu20.04' for 'mysql-community-server' was not found
E: Version '8.0.22-1ubuntu20.04' for 'mysql-client' was not found
E: Version '8.0.22-1ubuntu20.04' for 'mysql-common' was not found
E: Version '8.0.22-1ubuntu20.04' for 'mysql-workbench-community' was not found

Hi,

I need to double check but I don't think "8.0.22-1ubuntu20.04" are Oracle packages. This is a bug on the ubuntu side (that 8.0.22 is missing and that 8.0.23 did not upgrade properly too).

Are you using https://dev.mysql.com/downloads/repo/apt/ ?

Hi,

I doublechecked, you are using ubuntu repository with ubuntu provided packages. You need to report this bug to them as the upgrade and downgrade as well as package availability on their repo is something we cannot influence. So please report the bug to them: https://help.ubuntu.com/community/ReportingBugs

I strongly advise you migrate to Oracle provided binaries. You install APT repo from Oracle: https://dev.mysql.com/downloads/repo/apt/ and you setup mysql-community-server and mysql-community-client.

https://dev.mysql.com/doc/mysql-apt-repo-quick-guide/en/

Thanks
Bogdan