• Post Reply Bookmark Topic Watch Topic
  • New Topic
programming forums Java Mobile Certification Databases Caching Books Engineering Micro Controllers OS Languages Paradigms IDEs Build Tools Frameworks Application Servers Open Source This Site Careers Other Pie Elite all forums
this forum made possible by our volunteer staff, including ...
Marshals:
  • Campbell Ritchie
  • Ron McLeod
  • Rob Spoor
  • Tim Cooke
  • Junilu Lacar
Sheriffs:
  • Henry Wong
  • Liutauras Vilda
  • Jeanne Boyarsky
Saloon Keepers:
  • Jesse Silverman
  • Tim Holloway
  • Stephan van Hulst
  • Tim Moores
  • Carey Brown
Bartenders:
  • Al Hobbs
  • Mikalai Zaikin
  • Piet Souris

Java reading and output too slow

 
Ranch Hand
Posts: 50
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
I have a code below that is reading large image size and writing them to file however the process is too slow. how can i refine this code in such a way that it will work faster?

 
Rancher
Posts: 3742
16
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator

benjamin parker wrote:I have a code below that is reading large image size and writing them to file however the process is too slow.


Is it downloading of the file or writing it to disk that is taking up the time ?
There is no point in anyone offering a solution until we know where the problem is.
 
benjamin parker
Ranch Hand
Posts: 50
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
i don't think it is due to writing to file that is taking a long time. probably the downloading part?
 
Joanne Neal
Rancher
Posts: 3742
16
  • Likes 3
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator

benjamin parker wrote:i don't think it is due to writing to file that is taking a long time. probably the downloading part?


Think and probably are not good words in this situation. You need to know where the problem is.
Have you tried taking the code that writes to the file out, so that you can see how long just the download takes ?
If it does turn out to be the download, have you tried downloading the file from another client (e.g. a browser), so that you have some idea of how bad your applications performance is ? After all, the problem may be your internet connection or your computer hardware.
 
Rancher
Posts: 4801
50
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Try reading and writing using the byte[] version of read (ensuring you note the size of the data read each time).
Reading and writing a byte at a time is slow.

Make the buffer a few K in size (I usually use 8, for no better reason than I always have).
 
lowercase baba
Posts: 13002
66
Chrome Java Linux
  • Likes 2
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
"too slow" is also not a good spec. How fast should it be? What is acceptable and what is not? You would usually define some kind of through-put. I.e. saying "a file should download in one second" is no good, since file A might be 10k, but file B could be 2GB.

The first rule of optimization is to define what - EXACTLY - the requirements are, otherwise you are constantly being told "make it faster" regardless of what you do.

The next rule is to figure out where the time is ACTUALLY being spent. It does no good to optimize something to be 90% faster if it is only taking 1% of the overall time. Find the real, actual, proven bottlenecks, and focus on what you can do to improve THAT piece.
 
Ranch Hand
Posts: 499
Spring AngularJS Java
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
benjamin

Find out the problematic phase first as Neal says. If its downloading part, use a 4k buffer and byte[] to download instead of downloading it as int. IMO transferring a file as int is a very bad practice, so kindly avoid it. Also trying writing the files in byte[] format. If you want some examples

Real'sHowTo
Java2s

These examples are file transfer over sockets. You can just concentrate on the transferring part or If you are interested you can very well go ahead and try file transfer in sockets
 
Marshal
Posts: 73970
332
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
I think this question too difficult for this forum. I shall move it.
 
Dave Tolls
Rancher
Posts: 4801
50
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator

fred rosenberger wrote:"too slow" is also not a good spec. How fast should it be? What is acceptable and what is not? You would usually define some kind of through-put. I.e. saying "a file should download in one second" is no good, since file A might be 10k, but file B could be 2GB.



Though this is quite true, I will say that transferring from InStream to OutStream a byte at a time is a fairly basic error in this sort of thing.
I wouldn't consider changing this code from the current byte-in, byte-out mode to a byte[] mode to be premature optimisation. I would consider it something that a code review should bring up.
 
Bartender
Posts: 10780
71
Hibernate Eclipse IDE Ubuntu
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator

Dave Tolls wrote:Though this is quite true, I will say that transferring from InStream to OutStream a byte at a time is a fairly basic error in this sort of thing.
I wouldn't consider changing this code from the current byte-in, byte-out mode to a byte[] mode to be premature optimisation.


I would. Calling a Java method a million times might be a drop in the ocean compared to something like disk latency, since method calls take nanoseconds, and if raw transfer is what is required, then byte-for-byte calls may be the only way to go.

What I would agree with is that a code review may well be the right forum for working out where the bottleneck is occurring; but don't just assume that it's the "byte-for-byte" side of things just because it generates a lot of method calls.

Winston
 
Dave Tolls
Rancher
Posts: 4801
50
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator

Winston Gutkowski wrote:

Dave Tolls wrote:Though this is quite true, I will say that transferring from InStream to OutStream a byte at a time is a fairly basic error in this sort of thing.
I wouldn't consider changing this code from the current byte-in, byte-out mode to a byte[] mode to be premature optimisation.


I would. Calling a Java method a million times might be a drop in the ocean compared to something like disk latency, since method calls take nanoseconds, and if raw transfer is what is required, then byte-for-byte calls may be the only way to go.

What I would agree with is that a code review may well be the right forum for working out where the bottleneck is occurring; but don't just assume that it's the "byte-for-byte" side of things just because it generates a lot of method calls.

Winston



It's not because it generates a lot of method calls.
It's because a generates a lot of traffic.
It's a FOS, so disk write is invariably a problem.
Indeed, the read via the Http connection can be.

I don't think I've seen an instance where writing to a file byte for byte is not a mistake. Which is why I've mentioned it.
And it's such a simple change that (though I doubt it's the source of the slowness) a fix is minimal effort.
 
Winston Gutkowski
Bartender
Posts: 10780
71
Hibernate Eclipse IDE Ubuntu
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator

Dave Tolls wrote:It's not because it generates a lot of method calls.
It's because a generates a lot of traffic.
It's a FOS, so disk write is invariably a problem.


Possibly, but I don't think you can state that categorically. For example, do you know for certain the exact system call that takes place when you issue a write() call? I certainly don't, and I was an admin for 15 years. For all I know, on a modern disk, it could be writing to some non-volatile cache. Disks are very different beasts these days to the old clunkers we (or I) had to deal with. They're practically mini-OS's of their own.

I don't think I've seen an instance where writing to a file byte for byte is not a mistake. Which is why I've mentioned it.


Well I have. Or rather, I've seen the converse - problems caused by buffered output that was not completed in the event of a crash.

All I'm saying is that you need to be very careful before you make assumptions about what the problem is before you know all the facts.

Winston
 
The airline is called "Virgin"? Don't you want a plane to go all the way? This tiny ad will go all the way:
Thread Boost feature
https://coderanch.com/t/674455/Thread-Boost-feature
reply
    Bookmark Topic Watch Topic
  • New Topic