Bug #43071 Handshake problem on EBCDIC platforms
Submitted: 20 Feb 23:54 Modified: 26 Feb 18:19
Reporter: Todd Farmer
Status: Closed
Category:Connector/J Severity:S2 (Serious)
Version:5.1.7 OS:Any
Assigned to: Todd Farmer Target Version:

[20 Feb 23:54] Todd Farmer
Description:
Existing Java application connects fine to 4.0 server, but after upgrade to 5.0,
connection fails due to auth failure.  Debugging code was added to C/J, and it was
determined that the encryption seed being passed from the server was being mangled in
C/J.

We read the bytes into an ASCII-encoded String:

this.seed = buf.readString("ASCII", getExceptionInterceptor());

When we go to generate the password hash, we convert the String back to a byte array, but
we don't specify the encoding, allowing the system default encoding to be used:

byte[] seedAsBytes = seed.getBytes(); // for debugging

This causes seedAsBytes to have incorrect values on EBCDIC platforms, and the subsequent
hashes generated will cause authentication failures.

How to repeat:
Try to log in from a z/OS machine.  ;)

Suggested fix:
Change:

byte[] seedAsBytes = seed.getBytes(); // for debugging

to 

byte[] seedAsBytes = seed.getBytes("ASCII"); // for debugging
[21 Feb 0:16] Todd Farmer
Patch pushed (http://lists.mysql.com/commits/67071).
[26 Feb 18:19] Tony Bedford
An entry was added to the 5.1.8 changelog:

When the MySQL Server was upgraded from 4.0 to 5.0, the Connector/J application then
failed to connect to the server. This was because authentication failed when the
application ran from EBCDIC platforms such as z/OS.