• Post Reply Bookmark Topic Watch Topic
  • New Topic

Java reading and output too slow  RSS feed

 
benjamin parker
Ranch Hand
Posts: 50
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
I have a code below that is reading large image size and writing them to file however the process is too slow. how can i refine this code in such a way that it will work faster?

 
Joanne Neal
Rancher
Posts: 3742
16
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
benjamin parker wrote:I have a code below that is reading large image size and writing them to file however the process is too slow.

Is it downloading of the file or writing it to disk that is taking up the time ?
There is no point in anyone offering a solution until we know where the problem is.
 
benjamin parker
Ranch Hand
Posts: 50
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
i don't think it is due to writing to file that is taking a long time. probably the downloading part?
 
Joanne Neal
Rancher
Posts: 3742
16
  • Likes 3
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
benjamin parker wrote:i don't think it is due to writing to file that is taking a long time. probably the downloading part?

Think and probably are not good words in this situation. You need to know where the problem is.
Have you tried taking the code that writes to the file out, so that you can see how long just the download takes ?
If it does turn out to be the download, have you tried downloading the file from another client (e.g. a browser), so that you have some idea of how bad your applications performance is ? After all, the problem may be your internet connection or your computer hardware.
 
Dave Tolls
Ranch Hand
Posts: 2839
30
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Try reading and writing using the byte[] version of read (ensuring you note the size of the data read each time).
Reading and writing a byte at a time is slow.

Make the buffer a few K in size (I usually use 8, for no better reason than I always have).
 
fred rosenberger
lowercase baba
Bartender
Posts: 12529
48
Chrome Java Linux
  • Likes 2
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
"too slow" is also not a good spec. How fast should it be? What is acceptable and what is not? You would usually define some kind of through-put. I.e. saying "a file should download in one second" is no good, since file A might be 10k, but file B could be 2GB.

The first rule of optimization is to define what - EXACTLY - the requirements are, otherwise you are constantly being told "make it faster" regardless of what you do.

The next rule is to figure out where the time is ACTUALLY being spent. It does no good to optimize something to be 90% faster if it is only taking 1% of the overall time. Find the real, actual, proven bottlenecks, and focus on what you can do to improve THAT piece.
 
Partheban Udayakumar
Ranch Hand
Posts: 499
AngularJS Java Spring
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
benjamin

Find out the problematic phase first as Neal says. If its downloading part, use a 4k buffer and byte[] to download instead of downloading it as int. IMO transferring a file as int is a very bad practice, so kindly avoid it. Also trying writing the files in byte[] format. If you want some examples

Real'sHowTo
Java2s

These examples are file transfer over sockets. You can just concentrate on the transferring part or If you are interested you can very well go ahead and try file transfer in sockets
 
Campbell Ritchie
Sheriff
Posts: 55351
157
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
I think this question too difficult for this forum. I shall move it.
 
Dave Tolls
Ranch Hand
Posts: 2839
30
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
fred rosenberger wrote:"too slow" is also not a good spec. How fast should it be? What is acceptable and what is not? You would usually define some kind of through-put. I.e. saying "a file should download in one second" is no good, since file A might be 10k, but file B could be 2GB.


Though this is quite true, I will say that transferring from InStream to OutStream a byte at a time is a fairly basic error in this sort of thing.
I wouldn't consider changing this code from the current byte-in, byte-out mode to a byte[] mode to be premature optimisation. I would consider it something that a code review should bring up.
 
Winston Gutkowski
Bartender
Posts: 10573
65
Eclipse IDE Hibernate Ubuntu
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Dave Tolls wrote:Though this is quite true, I will say that transferring from InStream to OutStream a byte at a time is a fairly basic error in this sort of thing.
I wouldn't consider changing this code from the current byte-in, byte-out mode to a byte[] mode to be premature optimisation.

I would. Calling a Java method a million times might be a drop in the ocean compared to something like disk latency, since method calls take nanoseconds, and if raw transfer is what is required, then byte-for-byte calls may be the only way to go.

What I would agree with is that a code review may well be the right forum for working out where the bottleneck is occurring; but don't just assume that it's the "byte-for-byte" side of things just because it generates a lot of method calls.

Winston
 
Dave Tolls
Ranch Hand
Posts: 2839
30
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Winston Gutkowski wrote:
Dave Tolls wrote:Though this is quite true, I will say that transferring from InStream to OutStream a byte at a time is a fairly basic error in this sort of thing.
I wouldn't consider changing this code from the current byte-in, byte-out mode to a byte[] mode to be premature optimisation.

I would. Calling a Java method a million times might be a drop in the ocean compared to something like disk latency, since method calls take nanoseconds, and if raw transfer is what is required, then byte-for-byte calls may be the only way to go.

What I would agree with is that a code review may well be the right forum for working out where the bottleneck is occurring; but don't just assume that it's the "byte-for-byte" side of things just because it generates a lot of method calls.

Winston


It's not because it generates a lot of method calls.
It's because a generates a lot of traffic.
It's a FOS, so disk write is invariably a problem.
Indeed, the read via the Http connection can be.

I don't think I've seen an instance where writing to a file byte for byte is not a mistake. Which is why I've mentioned it.
And it's such a simple change that (though I doubt it's the source of the slowness) a fix is minimal effort.
 
Winston Gutkowski
Bartender
Posts: 10573
65
Eclipse IDE Hibernate Ubuntu
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Dave Tolls wrote:It's not because it generates a lot of method calls.
It's because a generates a lot of traffic.
It's a FOS, so disk write is invariably a problem.

Possibly, but I don't think you can state that categorically. For example, do you know for certain the exact system call that takes place when you issue a write() call? I certainly don't, and I was an admin for 15 years. For all I know, on a modern disk, it could be writing to some non-volatile cache. Disks are very different beasts these days to the old clunkers we (or I) had to deal with. They're practically mini-OS's of their own.

I don't think I've seen an instance where writing to a file byte for byte is not a mistake. Which is why I've mentioned it.

Well I have. Or rather, I've seen the converse - problems caused by buffered output that was not completed in the event of a crash.

All I'm saying is that you need to be very careful before you make assumptions about what the problem is before you know all the facts.

Winston
 
  • Post Reply Bookmark Topic Watch Topic
  • New Topic
Boost this thread!