Java reading and output too slow

[benjamin parker]

I have a code below that is reading large image size and writing them to file however the process is too slow. how can i refine this code in such a way that it will work faster?

import java.io.FileOutputStream;
import java.io.IOException;
import java.io.InputStream;
import java.net.MalformedURLException;
import java.net.URL;
import java.net.HttpURLConnection;

class TODOWNLOAD{

	public static void main(String[] args) {

	
		if (args.length != 2) {
			System.out.println("Arg: HostName Filename");
			System.exit(1);
		}
		
		// Configure URL address syntax
		String urlString = args[0].trim();	
		if (!urlString.startsWith("http://")) {
			urlString = "http://" + urlString;
		}
		
		String outputFileName = args[1].trim();

		try {		
			FileOutputStream outputStream = new FileOutputStream(outputFileName);

			URL urlAddress = new URL(urlString);
			HttpURLConnection urlConnection = (HttpURLConnection) urlAddress.openConnection();
			int responseCode = urlConnection.getResponseCode();
			
			if (responseCode == 200) {
				InputStream inputStream = urlAddress.openStream();
				
				int i = inputStream.read();
				while (i != -1) {
					outputStream.write(i);
					i = inputStream.read();
				}
				
				inputStream.close();
			} else {
				String responseCodeString = new Integer(responseCode).toString();
				byte[] responseCodeBytes = responseCodeString.getBytes();
				outputStream.write(responseCodeBytes);
			}
			
			outputStream.close();
		} catch (MalformedURLException e) {
			e.printStackTrace();
			System.exit(1);
		} catch (IOException e) {
			e.printStackTrace();
			System.exit(1);
		} catch (Exception e) {
			e.printStackTrace();
			System.exit(1);
		}
	}
}

[Joanne Neal]

benjamin parker wrote:I have a code below that is reading large image size and writing them to file however the process is too slow.

Is it downloading of the file or writing it to disk that is taking up the time ?
There is no point in anyone offering a solution until we know where the problem is.

[benjamin parker]

i don't think it is due to writing to file that is taking a long time. probably the downloading part?

[Joanne Neal]

benjamin parker wrote:i don't think it is due to writing to file that is taking a long time. probably the downloading part?

Think and probably are not good words in this situation. You need to know where the problem is.
Have you tried taking the code that writes to the file out, so that you can see how long just the download takes ?
If it does turn out to be the download, have you tried downloading the file from another client (e.g. a browser), so that you have some idea of how bad your applications performance is ? After all, the problem may be your internet connection or your computer hardware.

[Dave Tolls]

Try reading and writing using the byte[] version of read (ensuring you note the size of the data read each time).
Reading and writing a byte at a time is slow.

Make the buffer a few K in size (I usually use 8, for no better reason than I always have).

[fred rosenberger]

"too slow" is also not a good spec. How fast should it be? What is acceptable and what is not? You would usually define some kind of through-put. I.e. saying "a file should download in one second" is no good, since file A might be 10k, but file B could be 2GB.

The first rule of optimization is to define what - EXACTLY - the requirements are, otherwise you are constantly being told "make it faster" regardless of what you do.

The next rule is to figure out where the time is ACTUALLY being spent. It does no good to optimize something to be 90% faster if it is only taking 1% of the overall time. Find the real, actual, proven bottlenecks, and focus on what you can do to improve THAT piece.

[Partheban Udayakumar]

benjamin

Find out the problematic phase first as Neal says. If its downloading part, use a 4k buffer and byte[] to download instead of downloading it as int. IMO transferring a file as int is a very bad practice, so kindly avoid it. Also trying writing the files in byte[] format. If you want some examples

Real'sHowTo
Java2s

These examples are file transfer over sockets. You can just concentrate on the transferring part or If you are interested you can very well go ahead and try file transfer in sockets

[Campbell Ritchie]

I think this question too difficult for this forum. I shall move it.

[Dave Tolls]

fred rosenberger wrote:"too slow" is also not a good spec. How fast should it be? What is acceptable and what is not? You would usually define some kind of through-put. I.e. saying "a file should download in one second" is no good, since file A might be 10k, but file B could be 2GB.

Though this is quite true, I will say that transferring from InStream to OutStream a byte at a time is a fairly basic error in this sort of thing.
I wouldn't consider changing this code from the current byte-in, byte-out mode to a byte[] mode to be premature optimisation. I would consider it something that a code review should bring up.

[Winston Gutkowski]

Dave Tolls wrote:Though this is quite true, I will say that transferring from InStream to OutStream a byte at a time is a fairly basic error in this sort of thing.
I wouldn't consider changing this code from the current byte-in, byte-out mode to a byte[] mode to be premature optimisation.

I would. Calling a Java method a million times might be a drop in the ocean compared to something like disk latency, since method calls take nanoseconds, and if raw transfer is what is required, then byte-for-byte calls may be the only way to go.

What I would agree with is that a code review may well be the right forum for working out where the bottleneck is occurring; but don't just assume that it's the "byte-for-byte" side of things just because it generates a lot of method calls.

Winston

[Dave Tolls]

Winston Gutkowski wrote:

Dave Tolls wrote:Though this is quite true, I will say that transferring from InStream to OutStream a byte at a time is a fairly basic error in this sort of thing.
I wouldn't consider changing this code from the current byte-in, byte-out mode to a byte[] mode to be premature optimisation.

I would. Calling a Java method a million times might be a drop in the ocean compared to something like disk latency, since method calls take nanoseconds, and if raw transfer is what is required, then byte-for-byte calls may be the only way to go.

What I would agree with is that a code review may well be the right forum for working out where the bottleneck is occurring; but don't just assume that it's the "byte-for-byte" side of things just because it generates a lot of method calls.

Winston

It's not because it generates a lot of method calls.
It's because a generates a lot of traffic.
It's a FOS, so disk write is invariably a problem.
Indeed, the read via the Http connection can be.

I don't think I've seen an instance where writing to a file byte for byte is not a mistake. Which is why I've mentioned it.
And it's such a simple change that (though I doubt it's the source of the slowness) a fix is minimal effort.

[Winston Gutkowski]

Dave Tolls wrote:It's not because it generates a lot of method calls.
It's because a generates a lot of traffic.
It's a FOS, so disk write is invariably a problem.

Possibly, but I don't think you can state that categorically. For example, do you know for certain the exact system call that takes place when you issue a write() call? I certainly don't, and I was an admin for 15 years. For all I know, on a modern disk, it could be writing to some non-volatile cache. Disks are very different beasts these days to the old clunkers we (or I) had to deal with. They're practically mini-OS's of their own.

I don't think I've seen an instance where writing to a file byte for byte is not a mistake. Which is why I've mentioned it.

Well I have. Or rather, I've seen the converse - problems caused by buffered output that was not completed in the event of a crash.

All I'm saying is that you need to be very careful before you make assumptions about what the problem is before you know all the facts.

Winston