Bug #43071 Handshake problem on EBCDIC platforms
Submitted: 20 Feb 2009 22:54 Modified: 26 Feb 2009 17:19
Reporter: Todd Farmer (OCA) Email Updates:
Status: Closed Impact on me:
Category:Connector / J Severity:S2 (Serious)
Version:5.1.7 OS:Any
Assigned to: Todd Farmer CPU Architecture:Any

[20 Feb 2009 22:54] Todd Farmer
Existing Java application connects fine to 4.0 server, but after upgrade to 5.0, connection fails due to auth failure.  Debugging code was added to C/J, and it was determined that the encryption seed being passed from the server was being mangled in C/J.

We read the bytes into an ASCII-encoded String:

this.seed = buf.readString("ASCII", getExceptionInterceptor());

When we go to generate the password hash, we convert the String back to a byte array, but we don't specify the encoding, allowing the system default encoding to be used:

byte[] seedAsBytes = seed.getBytes(); // for debugging

This causes seedAsBytes to have incorrect values on EBCDIC platforms, and the subsequent hashes generated will cause authentication failures.

How to repeat:
Try to log in from a z/OS machine.  ;)

Suggested fix:

byte[] seedAsBytes = seed.getBytes(); // for debugging


byte[] seedAsBytes = seed.getBytes("ASCII"); // for debugging
[20 Feb 2009 23:16] Todd Farmer
Patch pushed (http://lists.mysql.com/commits/67071).
[26 Feb 2009 17:19] Tony Bedford
An entry was added to the 5.1.8 changelog:

When the MySQL Server was upgraded from 4.0 to 5.0, the Connector/J application then failed to connect to the server. This was because authentication failed when the application ran from EBCDIC platforms such as z/OS.