Forums Register Login

Java reading and output too slow

+Pie Number of slices to send: Send
I have a code below that is reading large image size and writing them to file however the process is too slow. how can i refine this code in such a way that it will work faster?

+Pie Number of slices to send: Send
 

benjamin parker wrote:I have a code below that is reading large image size and writing them to file however the process is too slow.


Is it downloading of the file or writing it to disk that is taking up the time ?
There is no point in anyone offering a solution until we know where the problem is.
+Pie Number of slices to send: Send
i don't think it is due to writing to file that is taking a long time. probably the downloading part?
3
+Pie Number of slices to send: Send
 

benjamin parker wrote:i don't think it is due to writing to file that is taking a long time. probably the downloading part?


Think and probably are not good words in this situation. You need to know where the problem is.
Have you tried taking the code that writes to the file out, so that you can see how long just the download takes ?
If it does turn out to be the download, have you tried downloading the file from another client (e.g. a browser), so that you have some idea of how bad your applications performance is ? After all, the problem may be your internet connection or your computer hardware.
+Pie Number of slices to send: Send
Try reading and writing using the byte[] version of read (ensuring you note the size of the data read each time).
Reading and writing a byte at a time is slow.

Make the buffer a few K in size (I usually use 8, for no better reason than I always have).
2
+Pie Number of slices to send: Send
"too slow" is also not a good spec. How fast should it be? What is acceptable and what is not? You would usually define some kind of through-put. I.e. saying "a file should download in one second" is no good, since file A might be 10k, but file B could be 2GB.

The first rule of optimization is to define what - EXACTLY - the requirements are, otherwise you are constantly being told "make it faster" regardless of what you do.

The next rule is to figure out where the time is ACTUALLY being spent. It does no good to optimize something to be 90% faster if it is only taking 1% of the overall time. Find the real, actual, proven bottlenecks, and focus on what you can do to improve THAT piece.
+Pie Number of slices to send: Send
benjamin

Find out the problematic phase first as Neal says. If its downloading part, use a 4k buffer and byte[] to download instead of downloading it as int. IMO transferring a file as int is a very bad practice, so kindly avoid it. Also trying writing the files in byte[] format. If you want some examples

Real'sHowTo
Java2s

These examples are file transfer over sockets. You can just concentrate on the transferring part or If you are interested you can very well go ahead and try file transfer in sockets
+Pie Number of slices to send: Send
I think this question too difficult for this forum. I shall move it.
+Pie Number of slices to send: Send
 

fred rosenberger wrote:"too slow" is also not a good spec. How fast should it be? What is acceptable and what is not? You would usually define some kind of through-put. I.e. saying "a file should download in one second" is no good, since file A might be 10k, but file B could be 2GB.



Though this is quite true, I will say that transferring from InStream to OutStream a byte at a time is a fairly basic error in this sort of thing.
I wouldn't consider changing this code from the current byte-in, byte-out mode to a byte[] mode to be premature optimisation. I would consider it something that a code review should bring up.
+Pie Number of slices to send: Send
 

Dave Tolls wrote:Though this is quite true, I will say that transferring from InStream to OutStream a byte at a time is a fairly basic error in this sort of thing.
I wouldn't consider changing this code from the current byte-in, byte-out mode to a byte[] mode to be premature optimisation.


I would. Calling a Java method a million times might be a drop in the ocean compared to something like disk latency, since method calls take nanoseconds, and if raw transfer is what is required, then byte-for-byte calls may be the only way to go.

What I would agree with is that a code review may well be the right forum for working out where the bottleneck is occurring; but don't just assume that it's the "byte-for-byte" side of things just because it generates a lot of method calls.

Winston
+Pie Number of slices to send: Send
 

Winston Gutkowski wrote:

Dave Tolls wrote:Though this is quite true, I will say that transferring from InStream to OutStream a byte at a time is a fairly basic error in this sort of thing.
I wouldn't consider changing this code from the current byte-in, byte-out mode to a byte[] mode to be premature optimisation.


I would. Calling a Java method a million times might be a drop in the ocean compared to something like disk latency, since method calls take nanoseconds, and if raw transfer is what is required, then byte-for-byte calls may be the only way to go.

What I would agree with is that a code review may well be the right forum for working out where the bottleneck is occurring; but don't just assume that it's the "byte-for-byte" side of things just because it generates a lot of method calls.

Winston



It's not because it generates a lot of method calls.
It's because a generates a lot of traffic.
It's a FOS, so disk write is invariably a problem.
Indeed, the read via the Http connection can be.

I don't think I've seen an instance where writing to a file byte for byte is not a mistake. Which is why I've mentioned it.
And it's such a simple change that (though I doubt it's the source of the slowness) a fix is minimal effort.
+Pie Number of slices to send: Send
 

Dave Tolls wrote:It's not because it generates a lot of method calls.
It's because a generates a lot of traffic.
It's a FOS, so disk write is invariably a problem.


Possibly, but I don't think you can state that categorically. For example, do you know for certain the exact system call that takes place when you issue a write() call? I certainly don't, and I was an admin for 15 years. For all I know, on a modern disk, it could be writing to some non-volatile cache. Disks are very different beasts these days to the old clunkers we (or I) had to deal with. They're practically mini-OS's of their own.

I don't think I've seen an instance where writing to a file byte for byte is not a mistake. Which is why I've mentioned it.


Well I have. Or rather, I've seen the converse - problems caused by buffered output that was not completed in the event of a crash.

All I'm saying is that you need to be very careful before you make assumptions about what the problem is before you know all the facts.

Winston
Blood pressure normal? What do I change to get "magnificent"? Maybe this tiny ad?
a bit of art, as a gift, the permaculture playing cards
https://gardener-gift.com


reply
reply
This thread has been viewed 5108 times.
Similar Threads
How to encode image in parts?
How to attach a certificate to a socket and send a https POST request?
Xml reading
Trying to Redirect. 302 Error. HTTP CLIENT
Read XML output from a URL
More...

All times above are in ranch (not your local) time.
The current ranch time is
Mar 28, 2024 23:09:31.