Bug #79783 Error Log misleading on Mysql and performance_schema errors when upgrading
Submitted: 28 Dec 2015 6:07 Modified: 28 Dec 2015 14:03
Reporter: Kasi Viswanadha Rao Jaladi Email Updates:
Status: Not a Bug Impact on me:
None 
Category:MySQL Server: Errors Severity:S3 (Non-critical)
Version:5.6.28 OS:CentOS (CentOS release 5.11 (Final))
Assigned to: CPU Architecture:Any
Tags: MySQL upgrade

[28 Dec 2015 6:07] Kasi Viswanadha Rao Jaladi
Description:
Hi,

I am doing a MySQL in place upgrade to latest 5.6.28 from 5.5.40 version. When started the MySQL server after latest RPM's installed (Before doing mysql_upgrade) , there are some errors logged in error log.

For example:

[ERROR] Column count of mysql.events_waits_current is wrong. Expected 19, found 16. Created with
 MySQL 50540, now running 50628. Please use mysql_upgrade to fix this error.

[ERROR] Column count of mysql.events_waits_history is wrong. Expected 19, found 16. Created with
 MySQL 50540, now running 50628. Please use mysql_upgrade to fix this error.

[ERROR] Column count of mysql.events_waits_history_long is wrong. Expected 19, found 16. Created
 with MySQL 50540, now running 50628. Please use mysql_upgrade to fix this error.

[ERROR] Column count of mysql.file_summary_by_event_name is wrong. Expected 23, found 5. Created
 with MySQL 50540, now running 50628. Please use mysql_upgrade to fix this error.

[ERROR] Column count of mysql.file_summary_by_instance is wrong. Expected 25, found 6. Created w
ith MySQL 50540, now running 50628. Please use mysql_upgrade to fix this error.

But All the tables mentioned above, do not belongs to mysql schema. 

mysql> show create table mysql.file_summary_by_instance;
ERROR 1146 (42S02): Table 'mysql.file_summary_by_instance' doesn't exist

mysql> select * from mysql.events_waits_history_long ;
ERROR 1146 (42S02): Table 'mysql.events_waits_history_long' doesn't exist

mysql> select * from mysql.events_waits_history ;
ERROR 1146 (42S02): Table 'mysql.events_waits_history' doesn't exist

mysql> select * from mysql.events_waits_current ;
ERROR 1146 (42S02): Table 'mysql.events_waits_current' doesn't exist

mysql> show create table mysql.file_summary_by_event_name ;
ERROR 1146 (42S02): Table 'mysql.file_summary_by_event_name' doesn't exist

They actually belongs to performance_schema.

mysql> \u performance_schema
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A

Database changed
mysql> show create table file_summary_by_instance;
| file_summary_by_instance | CREATE TABLE `file_summary_by_instance` (
  `FILE_NAME` varchar(512) NOT NULL,
  `EVENT_NAME` varchar(128) NOT NULL,
  `COUNT_READ` bigint(20) unsigned NOT NULL,
  `COUNT_WRITE` bigint(20) unsigned NOT NULL,
  `SUM_NUMBER_OF_BYTES_READ` bigint(20) unsigned NOT NULL,
  `SUM_NUMBER_OF_BYTES_WRITE` bigint(20) unsigned NOT NULL
) ENGINE=PERFORMANCE_SCHEMA DEFAULT CHARSET=utf8 |
1 row in set (0.00 sec)

mysql> show create table events_waits_history_long;
| events_waits_history_long | CREATE TABLE `events_waits_history_long` (
  `THREAD_ID` int(11) NOT NULL,
  `EVENT_ID` bigint(20) unsigned NOT NULL,
  `EVENT_NAME` varchar(128) NOT NULL,
  `SOURCE` varchar(64) DEFAULT NULL,
  `TIMER_START` bigint(20) unsigned DEFAULT NULL,
  `TIMER_END` bigint(20) unsigned DEFAULT NULL,
  `TIMER_WAIT` bigint(20) unsigned DEFAULT NULL,
  `SPINS` int(10) unsigned DEFAULT NULL,
  `OBJECT_SCHEMA` varchar(64) DEFAULT NULL,
  `OBJECT_NAME` varchar(512) DEFAULT NULL,
  `OBJECT_TYPE` varchar(64) DEFAULT NULL,
  `OBJECT_INSTANCE_BEGIN` bigint(20) NOT NULL,
  `NESTING_EVENT_ID` bigint(20) unsigned DEFAULT NULL,
  `OPERATION` varchar(16) NOT NULL,
  `NUMBER_OF_BYTES` bigint(20) unsigned DEFAULT NULL,
  `FLAGS` int(10) unsigned DEFAULT NULL
) ENGINE=PERFORMANCE_SCHEMA DEFAULT CHARSET=utf8 |
1 row in set (0.00 sec)

mysql> show create table events_waits_history;
| events_waits_history | CREATE TABLE `events_waits_history` (
  `THREAD_ID` int(11) NOT NULL,
  `EVENT_ID` bigint(20) unsigned NOT NULL,
  `EVENT_NAME` varchar(128) NOT NULL,
  `SOURCE` varchar(64) DEFAULT NULL,
  `TIMER_START` bigint(20) unsigned DEFAULT NULL,
  `TIMER_END` bigint(20) unsigned DEFAULT NULL,
  `TIMER_WAIT` bigint(20) unsigned DEFAULT NULL,
  `SPINS` int(10) unsigned DEFAULT NULL,
  `OBJECT_SCHEMA` varchar(64) DEFAULT NULL,
  `OBJECT_NAME` varchar(512) DEFAULT NULL,
  `OBJECT_TYPE` varchar(64) DEFAULT NULL,
  `OBJECT_INSTANCE_BEGIN` bigint(20) NOT NULL,
  `NESTING_EVENT_ID` bigint(20) unsigned DEFAULT NULL,
  `OPERATION` varchar(16) NOT NULL,
  `NUMBER_OF_BYTES` bigint(20) unsigned DEFAULT NULL,
  `FLAGS` int(10) unsigned DEFAULT NULL
) ENGINE=PERFORMANCE_SCHEMA DEFAULT CHARSET=utf8 |
1 row in set (0.00 sec)

mysql> show create table events_waits_current;
| events_waits_current | CREATE TABLE `events_waits_current` (
  `THREAD_ID` int(11) NOT NULL,
  `EVENT_ID` bigint(20) unsigned NOT NULL,
  `EVENT_NAME` varchar(128) NOT NULL,
  `SOURCE` varchar(64) DEFAULT NULL,
  `TIMER_START` bigint(20) unsigned DEFAULT NULL,
  `TIMER_END` bigint(20) unsigned DEFAULT NULL,
  `TIMER_WAIT` bigint(20) unsigned DEFAULT NULL,
  `SPINS` int(10) unsigned DEFAULT NULL,
  `OBJECT_SCHEMA` varchar(64) DEFAULT NULL,
  `OBJECT_NAME` varchar(512) DEFAULT NULL,
  `OBJECT_TYPE` varchar(64) DEFAULT NULL,
  `OBJECT_INSTANCE_BEGIN` bigint(20) NOT NULL,
  `NESTING_EVENT_ID` bigint(20) unsigned DEFAULT NULL,
  `OPERATION` varchar(16) NOT NULL,
  `NUMBER_OF_BYTES` bigint(20) unsigned DEFAULT NULL,
  `FLAGS` int(10) unsigned DEFAULT NULL
) ENGINE=PERFORMANCE_SCHEMA DEFAULT CHARSET=utf8 |
1 row in set (0.00 sec)

mysql> show create table file_summary_by_event_name;
file_summary_by_event_name | CREATE TABLE `file_summary_by_event_name` (
  `EVENT_NAME` varchar(128) NOT NULL,
  `COUNT_READ` bigint(20) unsigned NOT NULL,
  `COUNT_WRITE` bigint(20) unsigned NOT NULL,
  `SUM_NUMBER_OF_BYTES_READ` bigint(20) unsigned NOT NULL,
  `SUM_NUMBER_OF_BYTES_WRITE` bigint(20) unsigned NOT NULL
) ENGINE=PERFORMANCE_SCHEMA DEFAULT CHARSET=utf8 |

1 row in set (0.00 sec)

And we have lots of errors like below.

And lots of below error.
[ERROR] Native table 'performance_schema'.'setup_actors' has the wrong structure

But the table events_waits_summary_by_host_by_event_name does not exist at all in 5.5.40.

mysql> show create table performance_schema.setup_actors;
ERROR 1146 (42S02): Table 'performance_schema.setup_actors' doesn't exist

mysql> select version();
+------------+
| version()  |
+------------+
| 5.5.40-log |
+------------+
1 row in set (0.00 sec)

This table created in newer versions.

After doing mysql_upgrade and server restart , no errors (mentioned above) logged. 

How to repeat:
Shutdown MySQL 5.5.40 version server. Uninstall the old RPM's and install new 5.6.28 RPM's. Start the MySQL server.

Suggested fix:
1)The Error should point to correct schema(performance_schema) instead of wrong schema(mysql).

Old Error: Column count of mysql.events_waits_current is wrong. Expected 19, found 16. Created with
 MySQL 50540, now running 50628. Please use mysql_upgrade to fix this error.

Suggested Error: Column count of performance_schema.events_waits_current is wrong. Expected 19, found 16. Created with
 MySQL 50540, now running 50628. Please use mysql_upgrade to fix this error.

2)The error should mention like "Table does not exist" instead of logging that "Table has wrong structure".

Old Error:Native table 'performance_schema'.'table_io_waits_summary_by_index_usage' has the wrong structure

Suggested Error: Native table 'performance_schema'.'table_io_waits_summary_by_index_usage' does not exist.
[28 Dec 2015 13:56] MySQL Verification Team
Thank you for the bug report.

"After doing mysql_upgrade and server restart , no errors (mentioned above) logged."

https://dev.mysql.com/doc/refman/5.6/en/mysql-upgrade.html

"You should execute mysql_upgrade each time you upgrade MySQL. "

.......

"After running mysql_upgrade, stop the server and restart it so that any changes made to the system tables take effect. "
[28 Dec 2015 14:03] Kasi Viswanadha Rao Jaladi
I know mysql_upgrade do the required changes. But i am asking the error should be logged in a correct way and it should not misleading.