(HttpsServer) SSL extremely slow/times out from remote addresses, but not from local

[Rune Espensen]

Hi,


I am at a loss here - and I can't figure out whether it's my home network or my code that is stupid in this case :-)

Anyways, in short I have developed a small utility, that runs a HttpsServer for admin purposes. I have set it up with 2-way SSL (though, the problem is the same without the client auth). It all works (self-signed certificates etc.) when I run the java program and access through (any) browser from the local PC - no problem. But, as soon as I try to run it on a separate machine, and access the program "remotely" (e.g. from any other PC on my home network), the connection will time out, or sometimes succeed after several minutes and several retries.

Here is a list of stuff I have observed:

- The problem is only on SSL on port 443. If I run a normal HttpServer with no SSL, everything works as expected.
- I have tried different machines as servers with the java program on, and different client PC's (and cellphone) as clients (browser). The pattern is the same, it will not work.
- There's no firewall.
- I am using my ZyXel WiFi router
- It seems like the problem arises only when I hit the actual network. Localhost, loopback and local ip all works fine.
- There are no errors or exceptions being thrown server side (that I can see).
- If I try to access from a remote browser first, it will block requests from local browser as well (I guess it blocks until it finally fails on the handshake, before allowing new connections)
- I took at look at wireshark (unfortunately, I can't compare it to the "good" connections, as those don't hit the network), but it seems the initial SYN/ACK sequence, is OK. Then the client sends the Hello and the server just doesn't answer, or answers after the client has given up and closed the connection.
- The server keystore contains just the one selfsigned certificate (and also, the trusted client certificate - but again, the problem is the same with and without client auth)
- I would post code, but seems mute, as it works properly when local.
- I will try and find an old hub tomorrow ad set up a basic network...

Does anyone have experience with behavior like this? I have no idea on what to look at to continue debugging this, so any hints are welcome :-)

[Karthik Shiraly]

Hi Rune, a warm welcome to CodeRanch!

I don't have any answers, just a suggestion that you should mention the OS and HTTP server being used. That may ring a bell for somebody here who has experienced the same thing.

[Rune Espensen]

Hi,

Good idea :-)
It's on any OS I've tried (win7, raspian linux and server 2003)
The HttpServer is the Sun one from the javax package.

[Karthik Shiraly]

A localhost network capture using "tcpdump -i lo" on raspbian may help you compare.
I vaguely remember Microsoft's Network Monitor tool can capture localhost traffic (but I could be wrong about this, it's been a while and I'm not on Windows right now to confirm).

[Rune Espensen]

Good tip, Will try that :)


For good order, I am including the code here - pretty straightforward and nothing out of the ordinary I think:

static void main(String args[]){
		
		String keyStoreName = "store.jks";
		char keyStorePassword[] = "aPassword".toCharArray();
		char certPassword[] = "aPassword".toCharArray();

		try {
			System.setProperty("javax.net.ssl.trustStore", "store.jks");
			System.setProperty("javax.net.ssl.trustStorePassword", "aPassword");

			KeyStore keyStore = KeyStore.getInstance("JKS");
			keyStore.load(new FileInputStream(keyStoreName), keyStorePassword);
			KeyManagerFactory kFactory = KeyManagerFactory.getInstance("SunX509");
			kFactory.init(keyStore, certPassword);
			SSLContext sc = SSLContext.getInstance("SSL");
			sc.init(kFactory.getKeyManagers(), null, null);
			
			HttpsServer server = HttpsServer.create(new InetSocketAddress(443), 0);

			//Force client Auth
			HttpsConfigurator httpsConfigurator = new HttpsConfigurator(sc){
				public void configure (HttpsParameters params) {
			         SSLContext c = getSSLContext();
			         // get the default parameters
			         SSLParameters sslparams = c.getDefaultSSLParameters();
			        // sslparams.setNeedClientAuth(true);
			         params.setSSLParameters(sslparams);
			     }
			};
			
			server.setHttpsConfigurator(httpsConfigurator);
	        //Create contexts
			server.createContext("/", new HttpHandler(){

				@Override
				public void handle(HttpExchange t) throws IOException {
					System.out.println("Got request: "+t.getRequestURI());					
				}
				
			});
	        server.setExecutor(null); // creates a default executor
	        server.start();
	        System.out.println("\n******************************\n*  Admin Interface started.  *\n******************************");
		}catch(Exception e){
			System.out.println("Error starting Admin Interface");
			System.out.println(e.getClass()+"=>"+e.getMessage());
		}

		
	}

[Karthik Shiraly]

Please enclose your code snippets in code tags in future. I've done it for you this time.

I'm not familiar with Http(s)Server, but from javadocs of setExecutor(null) and start(), it looks like all requests are handled by a single background thread started by start().
This could explain atleast some of the observed behaviours, such as

"the connection will time out, or sometimes succeed after several minutes and several retries.".
"If I try to access from a remote browser first, it will block requests from local browser as well (I guess it blocks until it finally fails on the handshake, before allowing new connections) "
"client sends the Hello and the server just doesn't answer, or answers after the client has given up and closed the connection."

What I'm guessing is 1st connection gets delayed (for some reason as yet unknown, could simply be the SSL handshake time), then that blocks the second one which is in queue behind (because there is only 1 thread), which blocks the next one, and so on. Eventually the later ones in queue timeout. This delay may simply be the normal SSL handshake taking time and cascading the delays down the queue resulting in timeouts.
It's just a guess, but you can confirm if it's the case by setting setExecutor(Executors.newCachedThreadPool()) and see if successful requests increase a bit.

[Rune Espensen]

Excellent suggestion - makes total sense in regards to what I am seeing! Unfortunately, also makes no difference at all :/

I couldn't find my old hub, so I could not test out my network theory, so still at a loss here :-/

[Karthik Shiraly]

Ah, my guess was wrong then.
Did the tcpdump give any clue?
Another debugging option you can try out is run your server application with "-Djavax.net.debug=all" cmd line flag and see if it gives any clue. See http://docs.oracle.com/javase/7/docs/technotes/guides/security/jsse/ReadDebug.html

[Rune Espensen]

Finally, a breakthrough!

Ok, so I haven't had so much time the last week due to other projects, but I finally got some hours to spend on this.

Thanks for tipping me off on the debug trace for SSL, that helped a lot in tracking down the issue.

So, the culprit, is that upon receiving a secure request, the server will try to do a reverse DNS lookup for the client ip. Since there is no DNS to answer this request, it will wait for timeout - that takes about 20 seconds, but this lookup is made upon EACH request, making just the initial handshake take well over 1 minute.

This is also why it worked on the localhost, because it will resolve that through hosts file.

So, if I add the client IP to the hosts file on the server, it is down to a tolerable 1-2sec handshake (This is on a raspberry pi - so the encryption overhead is noticable, but manageble)

Now, the issue is of course, that I wont be able to maintain a full possible client list on the server, so I now need to figure out if it is possible to disable this reverse DNS check on the server. My guess is, that has to be done on the SSLSocket component, but that is not accesible through the HttpsServer and I can't seem to get to run a custom provider - so I am unsure on how to approach this or create a workaround. That is next step - right now I am just happy that I am no longer in the absolute dark as to why this is happening, so thank you for the feedback and suggestions! :-)

[Karthik Shiraly]

Good detective work, Rune! Thanks for posting, because I learnt some new things from your post.
So far, I hadn't realized there'd be DNS lookups during handshake. Thinking about it, perhaps it makes sense because certificates are usually associated with hostnames (though a search says CN=IP address is allowed but rare)
Anyway, I think the biggest learning for me is if I ever come across the symptom "local works, remote slow" in future, check the DNS.

As for a workaround, I'm not sure if disabling DNS is the answer (unless certificate is issued with IP address, in which case IP addresses have to be static).
In my network, the wifi router is acting as DNS server for the local network, and supports reverse DNS. It's running dnsmasq daemon.
Even if devices get different IPs every time, the router resolves local device hostnames to their currently allocated dynamic IPs, and vice versa.
That way, there is no need for each device to have list of other devices in its hosts file.

[Rune Espensen]

I just don't undersatand why my router is not responding to the DNS requests... perhaps I need to take a look at the configuration.

I don't think it is solved by adding the IP as CN in the client certificate. It is true, that the CN should be the hostname for the Server SSL, but for the Client certificate, it should not matter. Either way, the delay is before the client certificate is even sent (or even requested by the server), so it is on a lower level in the network layer. This is what makes me think I should be able to configure something on the SSLSocket itself... I need to dig deeper into that as soon as I find the time :-)