MySQL Bugs: #43071: Handshake problem on EBCDIC platforms

Bug #43071	Handshake problem on EBCDIC platforms
Submitted:	20 Feb 2009 22:54	Modified:	26 Feb 2009 17:19
Reporter:	Todd Farmer (OCA)	Email Updates:
Status:	Closed	Impact on me:	None
Category:	Connector / J	Severity:	S2 (Serious)
Version:	5.1.7	OS:	Any
Assigned to:	Todd Farmer	CPU Architecture:	Any

Description:
Existing Java application connects fine to 4.0 server, but after upgrade to 5.0, connection fails due to auth failure.  Debugging code was added to C/J, and it was determined that the encryption seed being passed from the server was being mangled in C/J.

We read the bytes into an ASCII-encoded String:

this.seed = buf.readString("ASCII", getExceptionInterceptor());

When we go to generate the password hash, we convert the String back to a byte array, but we don't specify the encoding, allowing the system default encoding to be used:

byte[] seedAsBytes = seed.getBytes(); // for debugging

This causes seedAsBytes to have incorrect values on EBCDIC platforms, and the subsequent hashes generated will cause authentication failures.

How to repeat:
Try to log in from a z/OS machine.  ;)

Suggested fix:
Change:

byte[] seedAsBytes = seed.getBytes(); // for debugging

to 

byte[] seedAsBytes = seed.getBytes("ASCII"); // for debugging

Patch pushed (http://lists.mysql.com/commits/67071).

An entry was added to the 5.1.8 changelog:

When the MySQL Server was upgraded from 4.0 to 5.0, the Connector/J application then failed to connect to the server. This was because authentication failed when the application ran from EBCDIC platforms such as z/OS.