MySQL Bugs: #114707: collect_diagnostics() fails with AttributeError when handling None diff values

Bug #114707	collect_diagnostics() fails with AttributeError when handling None diff values
Submitted:	19 Apr 2024 15:16	Modified:	10 Jun 2024 8:38
Reporter:	Ioannis Androulidakis (OCA)	Email Updates:
Status:	Closed	Impact on me:	None
Category:	Shell General / Core Client	Severity:	S3 (Non-critical)
Version:	8.3.0	OS:	CentOS (CentOS Stream 8)
Assigned to:		CPU Architecture:	x86 (Intel(R) Xeon(R) Silver 411)
Tags:	debug, diagnostics, group replicaiton, mysqlsh, python

Description:
At Booking.com we recently faced a MySQL outage related to Group Replication that we have already reported in SR 3-35210776191. During this outage we attempted to collect diagnostics [1] from all members of the GR cluster using the following command in an interactive MySQL shell:

util.debug.collect_diagnostics("MysqlDiagnosticsFullCluster.zip", {"allMembers": True})

Unfortunately, in one GR cluster member (former primary of the broken GR cluster) the above command failed with the following error:

AttributeError: 'NoneType' object has no attribute 'ljust'

(You can see more information about the error in the attached stack backtrace).

Looking into this a bit, we found out that the `dump_diff()` function [2] does not check for None values when it enumerates diff values and before it appends them to the current line [3].

[1] https://dev.mysql.com/doc/mysql-shell/8.3/en/mysql-shell-utilities-debug-diagnostics.html
[2] https://github.com/mysql/mysql-shell/blob/master/python/plugins/debug/collect_diagnostics....
[3] https://github.com/mysql/mysql-shell/blob/master/python/plugins/debug/collect_diagnostics....

How to repeat:
Reproducing the exact environment and scenario, i.e., the GR failure described in SR 3-35210776191, that caused the diagnostics debug utility to fail is non-trivial.

It looks like the diagnostics debug utility will emit the same error in different environments and scenarios, as long as `dump_diff()` [1] is invoked with a `diff` object that contains None diff values. 

[1] https://github.com/mysql/mysql-shell/blob/master/python/plugins/debug/collect_diagnostics....

Suggested fix:
Make the `dump_diff()` function more robust by adding a check for None diff values, before left-justifying them and appending them to the current line. 

Also, print a descriptive message to let the user know that a None diff value was detected and that this diff value will be skipped.

Text file with stacktrace

Attachment: collect-diagnostics-stacktrace.txt (text/plain), 925 bytes.

Thank you very much for your patch contribution, we appreciate it!

In order for us to continue the process of reviewing your contribution to MySQL, please send us a signed copy of the Oracle Contributor Agreement (OCA) as outlined in https://oca.opensource.oracle.com

Signing an OCA needs to be done only once and it's valid for all other Oracle governed Open Source projects as well.

Getting a signed/approved OCA on file will help us facilitate your contribution - this one, and others in the future.  

Please let me know, if you have any questions.

Thank you for your interest in MySQL.

> In order for us to continue the process of reviewing your contribution to MySQL, please send us a signed copy of the Oracle Contributor Agreement (OCA) as outlined in https://oca.opensource.oracle.com

Hi there, thanks for the headsup. 

I signed the Oracle Contributor Agreement (OCA) using my GitHub handle (ioandr) and Oracle account (androulidakis.ioannis@gmail.com) on Apr 22nd but have received no update/response since then.

Attempting to re-sign it fails with:

> You cannot submit a new request unless the one that is under review is approved or rejected or you withdraw your request.

I can always share with you the signed PDF over e-mail, please specify an e-mail address in that case.

Hi,

Thanks for the update, I'll get the data to relevant party :)

Thanks for the patch

Contribution submitted via Github - Handle None diff values when collecting diagnostics (*) Contribution by Ioannis Androulidakis (Github ioandr, mysql-shell/pull/20#issuecomment-2124223248): I confirm the code being submitted is offered under the terms of the OCA, and that I am authorized to contribute it.

Contribution: git_patch_1831849410.txt (text/plain), 1.71 KiB.

Hi there, any updates on this? 

It's rather small patch, let me know if you need anything from my side to proceed with review/merge.

Hi,
Nothing else from your side needed. Thanks for your contribution. We expect it to be published soon.

Posted by developer:
 
Added the following note to the MySQL Shell 8.4.1 and 9.0.0 release notes:
  
 util.collect_diagnostics() failed with an AttributeError when run against an InnoDB Cluster 
 on which the Group Replication plugin was uninstalled from one or more members. 
 The utility attempted to retrieve values for Group Replication system variables which did not exist because the plugin was uninstalled.

Thanks to Ioannis Androulidakis for the contribution.