Disabling handshake hello message in Java 1.6

[Scott Selikoff]

I have code that uses a HttpsURLConnection to perform a 2-way SSL handshake. It creates and uses an SSL Socket factory as follows:

final HttpsURLConnection urlc = (HttpsURLConnection) url.openConnection();
// Set properties of connection...
final SSLContext context = SSLContext.getInstance("TLS");
context.init(kmf.getKeyManagers(), tmf.getTrustManagers(), new SecureRandom());			
			
final SSLSocketFactory factory = context.getSocketFactory();
urlc.setSSLSocketFactory(factory);

The problem I'm having is that in Java 1.7 the code works fine, but in 1.6 it breaks with handshake error. After a lot of digging and stack traces I've isolated it to an "extra" handshake performed by Java 1.6. Specifically, Java 1.6 performs the following:

main, WRITE: TLSv1 Handshake, length = 75
main, WRITE: SSLv2 client hello message, length = 101
main, READ: TLSv1 Alert, length = 2
main, RECV TLSv1 ALERT:  fatal, handshake_failure

In Java 1.7, "SSLv2 client hello message" is never called. I've further isolated this to the documentation for JSSE which is on point with "Cause: Some SSL/TLS servers will disconnect if a ClientHello message is received in a format it doesn't understand or with a protocol version number that it doesn't support.".

The problem I'm having is the only way I seem to be able to disable the extra code is by calling "System.setProperty("https.protocols", "TLSv1");". This is a server environment though, and setting that parameter could have side effects that would take a lot of time to test. I haven't found another way, though. There is a SSLSocket.setEnabledProtocols() method, but the problem is HttpsURLConnection takes a factory, not a socket. I've tried extending my own SSL Socket factory with very little luck. Any suggestions?

[Jaikiran Pai]

Scott Selikoff wrote:There is a SSLSocket.setEnabledProtocols() method, but the problem is HttpsURLConnection takes a factory, not a socket. I've tried extending my own SSL Socket factory with very little luck. Any suggestions?

I think using a custom implementation of SSLSocketFactory should work, but the trick will be to use it only for overriding the enabled protocols and letting the usual SSLSocketFactory do the rest of the job. So something like:

final HttpsURLConnection urlc = (HttpsURLConnection) url.openConnection();  
// Set properties of connection...  
final SSLContext context = SSLContext.getInstance("TLS");  
context.init(kmf.getKeyManagers(), tmf.getTrustManagers(), new SecureRandom());           
              
// get hold of the real underlying socket factory which is expected to do the real job
final SSLSocketFactory factory = context.getSocketFactory();  
// "wrap" it around using our custom SSLSocketFactory so that we have control over the enabled protocols of newly created sockets
final SSLSocketFactory wrappedFactory = new ProtocolOverridingSSLSocketFactory(factory, new String[] {"TLSv1"});
// now pass the wrapped factory to the URL connection
urlc.setSSLSocketFactory(wrappedFactory); 

The custom SSLSocketFactory implementation will then look like:

package org.myapp;

import javax.net.ssl.SSLSocket;
import javax.net.ssl.SSLSocketFactory;
import java.io.IOException;
import java.net.InetAddress;
import java.net.Socket;
import java.net.UnknownHostException;

/**
 * A {@link SSLSocketFactory} which uses an existing {@link SSLSocketFactory} to delegate its operations to and overrides the
 * {@link javax.net.ssl.SSLSocket#getEnabledProtocols() enabled protocols} to the protocols that were passed to its
 * {@link #ProtocolOverridingSSLSocketFactory(javax.net.ssl.SSLSocketFactory, String[]) constructor}
 *
 * @author Jaikiran Pai
 */
public class ProtocolOverridingSSLSocketFactory extends SSLSocketFactory {

    private final SSLSocketFactory underlyingSSLSocketFactory;
    private final String[] enabledProtocols;

    public ProtocolOverridingSSLSocketFactory(final SSLSocketFactory delegate, final String[] enabledProtocols) {
        this.underlyingSSLSocketFactory = delegate;
        this.enabledProtocols = enabledProtocols;
    }

    @Override
    public String[] getDefaultCipherSuites() {
        return underlyingSSLSocketFactory.getDefaultCipherSuites();
    }

    @Override
    public String[] getSupportedCipherSuites() {
        return underlyingSSLSocketFactory.getSupportedCipherSuites();
    }

    @Override
    public Socket createSocket(final Socket socket, final String host, final int port, final boolean autoClose) throws IOException {
        final Socket underlyingSocket = underlyingSSLSocketFactory.createSocket(socket, host, port, autoClose);
        return overrideProtocol(underlyingSocket);
    }

    @Override
    public Socket createSocket(final String host, final int port) throws IOException, UnknownHostException {
        final Socket underlyingSocket = underlyingSSLSocketFactory.createSocket(host, port);
        return overrideProtocol(underlyingSocket);
    }

    @Override
    public Socket createSocket(final String host, final int port, final InetAddress localAddress, final int localPort) throws IOException, UnknownHostException {
        final Socket underlyingSocket = underlyingSSLSocketFactory.createSocket(host, port, localAddress, localPort);
        return overrideProtocol(underlyingSocket);
    }

    @Override
    public Socket createSocket(final InetAddress host, final int port) throws IOException {
        final Socket underlyingSocket = underlyingSSLSocketFactory.createSocket(host, port);
        return overrideProtocol(underlyingSocket);
    }

    @Override
    public Socket createSocket(final InetAddress host, final int port, final InetAddress localAddress, final int localPort) throws IOException {
        final Socket underlyingSocket = underlyingSSLSocketFactory.createSocket(host, port, localAddress, localPort);
        return overrideProtocol(underlyingSocket);
    }

    /**
     * Set the {@link javax.net.ssl.SSLSocket#getEnabledProtocols() enabled protocols} to {@link #enabledProtocols} if the <code>socket</code> is a
     * {@link SSLSocket}
     *
     * @param socket The Socket
     * @return
     */
    private Socket overrideProtocol(final Socket socket) {
        if (socket instanceof SSLSocket) {
            if (enabledProtocols != null && enabledProtocols.length > 0) {
                ((SSLSocket) socket).setEnabledProtocols(enabledProtocols);
            }
        }
        return socket;
    }


}

I haven't given it a try but I think it should work.

[Jaikiran Pai]

Scott Selikoff wrote:
The problem I'm having is that in Java 1.7 the code works fine, but in 1.6 it breaks with handshake error. After a lot of digging and stack traces I've isolated it to an "extra" handshake performed by Java 1.6.

For those curious as to why this works fine in Java 1.7, here's the reason from the Java 7 security enhancements/changes http://docs.oracle.com/javase/7/docs/technotes/guides/security/enhancements-7.html:

Changes in Java 7 wrote:
SSLv2Hello disabled by default on the client: In Java SE 7, SSLv2Hello is removed from the default enabled protocol list on the client.

[Scott Selikoff]

Beautiful wrapper implementation Jaikiran! Works perfectly. Thank you so much!

[Jaikiran Pai]

You're welcome

[Abdul Wahid]

Scoot and Jaikiran. I am also facing the similar issue. Can you please tell me whether we have an option to start the tomcat using TLSv1 in client communication rather than changing the entire code or upgrade the new JDK version.[1.7]

Thanks,
Abdul Wahid