nio download problem

[Samuel Mendenhall]

I am using nio to download a file given from a web address. I have not conducted extensive testing but my code works for text files and zip files, but NOT exe or bin files. I assume it works for all text and no binary (although zip is binary right?). The weird thing is, I set up tomcat, and I can download a zip, open the exe in it and it works, but if I extract the exe and try to download via tomcat, it downloads the correct number of bytes, but the exe is not valid, won't open.

I think my problem exists in the GET header I am passing the socket, maybe I need to define the mime type I want to accept? Any body can pass me some help, I'd be greatly appreciative.

Here is the GET header as descibed below also

String request = "GET " + "/files2/Atmn-Anywhere-setup200.exe" + " HTTP/1.0\r\n\r\n";

This is a full "working" example, the exe I am downloading I choose at random, choose another if you'd like.

package extractor;
 
import java.io.File;
import java.io.FileOutputStream;
import java.io.IOException;
import java.net.InetSocketAddress;
import java.net.URL;
import java.nio.ByteBuffer;
import java.nio.CharBuffer;
import java.nio.channels.SocketChannel;
import java.nio.channels.WritableByteChannel;
import java.nio.charset.Charset;
import java.nio.charset.CharsetEncoder;
 
public class Test {
public static void main(String[] args) {
String localDirectory = "C:\\java";
String remoteFilename = "Atmn-Anywhere-setup200.exe";
String address = "http://first.tucows.com/files2/Atmn-Anywhere-setup200.exe";
SocketChannel channel = null;
try {
URL url = new URL(address);

String host = url.getHost();
int port = url.getPort();
if(port == -1) {
port = 80;
}

// Setup
WritableByteChannel fileChannel = new FileOutputStream(localDirectory + 
                           File.separator + remoteFilename).getChannel();
InetSocketAddress socketAddress = new InetSocketAddress(host, port);
Charset charset = Charset.forName("ISO-8859-1");
CharsetEncoder encoder = charset.newEncoder();
 
// Allocate buffers
ByteBuffer buffer = ByteBuffer.allocateDirect(1024);
 
// Connect
channel = SocketChannel.open();
channel.connect(socketAddress);
channel.configureBlocking(false);

// Send request
String request = "GET " + "/files2/Atmn-Anywhere-setup200.exe" + " HTTP/1.0\r\n\r\n";
channel.write(encoder.encode(CharBuffer.wrap(request)));
 
// Read response
while ((channel.read(buffer)) != -1) {
buffer.flip();

//Write the buffer to the specified file also
fileChannel.write(buffer);

//Clear the buffers
buffer.clear();
}
} catch (Exception e) {
System.err.println(e);
}  finally {
if (channel != null) {
try {
channel.close();
} catch (IOException ignored) {
}
}
}
}
}
 

[ November 03, 2005: Message edited by: Joe Ess ]

[Samuel Mendenhall]

I think I partially figured out the problem. With the GET header I posted in the initial topic, I examined the response in ultraedit and the server said no host was defined, so here is my new GET

String request = "GET " + "/files2/Atmn-Anywhere-setup200.exe" + " HTTP/1.1\r\nHost: first.tucows.com\r\n\r\n";

Which seems to return the actual binary.....with the response Prepended.

So now my question is this, how do I rip the response out of the binary, while keeping the binary intact.

[Samuel Mendenhall]

New problem....

The above code is run in a thread, when the jvm exits out, and I try to delete the newly written file, Windows won't allow me, There is still a handle somewhere to the file.

I am closing the fileChannel however, what else should I do so there are no more handles/hooks/whatever attached to the file.

I do NOT have this problem with regular io, only nio.

[Joe Ess]

Originally posted by Samuel Mendenhall:

I do NOT have this problem with regular io, only nio.

Any reason you aren't using regular io?
Or for that matter, using sockets instead of URL/URLConnection?

[Samuel Mendenhall]

I might be misled, so please call me on it if I am; however I was led to believe that nio is faster because it uses native system calls, it is non blocking, and I am using the sockets becuase they should have less overhead than regular io.

From visual observance, nio seems to be much quicker than regular io.

[Samuel Mendenhall]

Additionally, I fixed my original problem of parsing out the HTTP response; I had to read and discard the bytes until ODOAODOA which is two newlines, which signifies the start of the content.

[Joe Ess]

You were misled. Regular IO and sockets were reimplemented using Channels when NIO was introduced so throughput differences should be minimal. Where they differ is in feature sets. NIO can be nonblocking (which is good for high-performance servers, doesn't do much for clients) and has other features like file locking. Have a look at Top Ten New Things in NIO for more on NIO.
Have a look at this example of downloading from a URL. You'll have to change it to use a Stream to handle binary files but it should give you an idea of how simple it is to use the existing classes rather reinventing the wheel (remember, being a productive programmer is as important as writing efficient code).

[Samuel Mendenhall]

Isn't there a performance hit on using streams instead of buffers?

[Jim Yingst]

NIO can be faster than IO, but in many situations there's not an appreciable difference, and sometimes IO is faster. Using buffers can indeed be faster than streams, as with buffers dont need to spend as much time passing bytes in and out of the JVM memory space. Especially if your Java program doesn't insist on actually looking at the individual bytes itself. You can often get really good performance from transferFrom() and transferTo() in FileChannel. To use this you need to know the number of bytes being transferred though - you should be able to get this from the response header. Note that you need to check the return value of these methods and put them in a loop to ensure that you've transferred everything you requested.

As far as NIO supporting nonblocking operations - that's true, but from the way you're using it I don't see any particular benefit to that. You've got a single thread dedicated to downloading one particular file, and if some of the content is not yet available from the server, there's nothing else for your program to do other than try again. You might as well just use blocking mode. If at some point in the future you find you have too many threads and want to instead have one thread handle multiple IO operations, then look into using nonblocking IO in conjunction with a Selector. For now though that seems like excessive complexity, in my opinion.

[Jim Yingst]

As far as being able to delete the file - you say you're closing the FileChannel, but your code above definitely does not close the FileChannel. Has the code been modified? And are you sure the FileChannel is really closed? From the way you close the other channel, it looks like if an error were to occur, you would have no evidence of it - the exception is ignored. I recommend at least putting a printStackTrace() or some more sophisticated logging into the catch block.

As an experiment, you might also insert a brief Thread.sleep() after the file close, before the delete attempt. It seems that on some [defective] systems, a file may not be fully "closed" yet when close() returns. See if adding a delay makes a difference.

[Samuel Mendenhall]

Thank you for the response.

I will be downloading potentially very large files, 100mb, 500, an arbitrarily large number and I will have multiples threads connecting to servers.


Here is some working code so far, it doesn't seem too complex to me...yet.

                SocketChannel channel = null;
WritableByteChannel fileChannel = null;
try {
URL url = new URL(address);
 
String host = url.getHost();
int port = url.getPort();
if (port == -1) {
port = 80;
}
 
// Setup
fileChannel = new FileOutputStream(localDirectory + File.separator
+ remoteFilename).getChannel();
InetSocketAddress socketAddress = new InetSocketAddress(host, port);
Charset charset = Charset.forName("ISO-8859-1");
// Charset charset = Charset.forName("US-ASCII");
CharsetEncoder encoder = charset.newEncoder();
// CharsetDecoder charDecorder = charset.newDecoder();
 
// Allocate buffers
ByteBuffer buffer = ByteBuffer.allocateDirect(1024);
// CharBuffer charbuf = CharBuffer.allocate(1024);
 
// Connect
channel = SocketChannel.open();
channel.connect(socketAddress);
channel.configureBlocking(false);
 
// Send request
String relativeRemotePath = getRelativeRemotePath(address);
String request = "GET " + relativeRemotePath + " HTTP/1.0\r\n\r\n";
channel.write(encoder.encode(CharBuffer.wrap(request)));
 
// Read Response
int numRead = 0;
boolean hasHTTPHeader = false;
boolean HTTPHeaderRemoved = false;
boolean first = true;
while (numRead >= 0) {
// read() places read bytes at the buffer's position so the
// position should always be properly set before calling read()
// This method sets the position to 0
buffer.rewind();
 
// Read bytes from the channel
numRead = channel.read(buffer);
 
// The read() method also moves the position so in order to
// read the new bytes, the buffer's position must be set back to
// 0
buffer.rewind();
 
// Read bytes from ByteBuffer; see also
// e159 Getting Bytes from a ByteBuffer
if (first) {
if (numRead > 4) {
byte[] bts = { buffer.get(), buffer.get(),
buffer.get(), buffer.get() };
numRead -= 4;
// bts = buffer.get(bts, 0, 4).array();
String probableHTTP = toHex(bts);
//Hex below stands for HTTP
hasHTTPHeader = probableHTTP.equals("0x48545450");
}
}
//TODO could throw an exception!
if(hasHTTPHeader && !HTTPHeaderRemoved) {
for (int i=0; i<numRead; i++) {
                if(toHex(buffer.get()).equals("0x0d") 
                && toHex(buffer.get()).equals("0x0a")
                && toHex(buffer.get()).equals("0x0d")
                && toHex(buffer.get()).equals("0x0a")) {
                HTTPHeaderRemoved = true;
                break;
                }
            }
}
fileChannel.write(buffer);
buffer.compact();
first = false;
}
} catch (Exception e) {
e.printStackTrace();
} finally {
if (channel.isOpen() || channel != null) {
try {
channel.close();
} catch (IOException ignored) {
ignored.printStackTrace();
}
}
if (fileChannel.isOpen() || fileChannel != null) {
try {
fileChannel.close();
} catch (IOException ignored) {
ignored.printStackTrace();
}
}
}

[Joe Ess]

One quick word, Reverse these two:

 if (channel.isOpen() || channel != null)  

If fileChannel is null you'll get a NullPointerException. Same with the other if statement on fileChannel.

[Samuel Mendenhall]

I'll do that, I can probably drop the .isOpen() and only test for !=null. I put the .isOpen in as a longshot becuase of the channel not closing on the file.