• Post Reply Bookmark Topic Watch Topic
  • New Topic

Chunk Size  RSS feed

 
Drew Lane
Ranch Hand
Posts: 296
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
When I am reading a byte array from a stream is there an optimal chunk size to use?
For example, would it be best to use chunks in increments of 256 or some other magic number?
//pseudo code
//read byte array in loop
{
byte[] buf = new byte[256];
dis.read(buf);
}
Does size matter?
Thanks,
Drew
 
qu qu
Greenhorn
Posts: 1
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
hi,
The optimal chunks size to use like this.
For example:

public String getStringFromfile(String str){
FileOutputStream output = new FileOutputStream(new File(str));
BufferedOutputStream buffered = new BufferedOutputStream(output);
byte[] size = new byte[buffered.available()];
buffered.read(size);
buffered.flush();
buffered.close();
String newString = new String(size);
return newString;
}
Do you know now?
if you have problem, you can send E-mail to me.
my E-mail:zhangquan0718@yahoo.com
 
Jim Yingst
Wanderer
Sheriff
Posts: 18671
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
"qu qu", please take a look at our display name policy and edit your display name to look like a real name. "Zhang Quan" would be fine, for example.
The code example using available() is extremely unreliable. That is, it will quite possibly work correctly in many situations, especially when the total amount of stream content is small. However the API for available() makes it clear that it may return a number less than the total number of bytes in the stream - it may even return 0. This means that there are 0 bytes available right now, but more may be available soon. There may be network delays, or time required for a disk drive's reader to reposition itself to another part of the disk (especially for a fragmented file). Do not rely on available() to determine the total size of stream content.
Drew - I don't think there is any standard answer to your question; the exact optimum value for a buffer size will probably vary on different machines, for different applications. You can conduct some performance test using different sizes to see what works best for your application. You will probably find that you can vary the value quite a bit without significantly affecting performance. I remember testing this a few years ago, and I found that when buffer size was in the tens or hundread, increasing buffer size had a noticable affect; once the buffer size was in the thousands, subsequent increases would have little effect. So for me, on my system at the time, 1024 seemed like a pretty good choice. (I use powers of two out of habit; maybe it's beneficial, maybe not.) However other systems may well benefit from more buffering. Nowadays I typically use 4096 or 8192 - I've seen other code using this (including in the core Java libararies) and figured, why not? Typically I'd declare it as a constant in whatever file uses it:

Makes it easier to spot and change later if you find it necessary.
Also, consider your total memory usage. For most applications, allocating a buffer of 8192 bytes is nothing. But if you've got a server with hundreds of threads, and each one manages to create its own buffer - well, 1 MB or more of memory is probably still not a big deal. But if might be. And if you replace 8192 with something larger, or have thousands of threads rather than hundread - you could be asking for trouble. And if you're programming for J2ME or something, you may have a very different idea of how much memory is "too much". So depending on your situation, you may find a completely different number is appropriate for buffer size.
 
Drew Lane
Ranch Hand
Posts: 296
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Thanks for the answer, Jim.
I just tried the available() method and it returned zero, so I think I'm gonna skip using that as you suggested.
I am working with memory constrained devices so I think I will try to keep the buffer small, maybe 512 or 1024.
I too, noticed a big performance improvement going from reading very small chunks to somewhat larger ones.
I was mostly curious about the optimal size, I'm not sure it makes a big difference at a certain point, either.
Regards,
Drew
 
  • Post Reply Bookmark Topic Watch Topic
  • New Topic
Boost this thread!