Win a copy of Functional Reactive Programming this week in the Other Languages forum!
  • Post Reply
  • Bookmark Topic Watch Topic
  • New Topic

Data Truncated when converting from blob to stream to char buffer for huge files

 
aadhar sharma
Ranch Hand
Posts: 38
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
I am loosing data when I try to convert a stream to a CharBuffer. The following can be verified from the output below.

I see the code perfectly fine and the following works 99% of the times when the file size is small how ever the data is truncated for huge files. (3,50,000 Lines approx or more)

The behaviour is consistent and the file is cut off every single time from a specific character.

Do the CharBuffer and ByteBuffer have a limitation on the size ? If yes is there a better way to re-write this code and return CharBuffer.



Any help appreciated.


InputStreamReader reader = null;
CharBuffer cbuf = null;
// get the blob to access its length
Blob blob = getBlob(attachment);
int capacity = (int)blob.length();
System.out.println("Capacity is this "+capacity);
System.out.println("Capacity is this long"+blob.length());
Log.info(this, "BinaryAttachmentManager getBufferredReader() allocating "+capacity+" buffer with characterset "+charsetName);
cbuf = CharBuffer.allocate(capacity);

BufferedInputStream bis = new BufferedInputStream(blob.getBinaryStream());
if(null!=charsetName) {
reader = new InputStreamReader(bis,charsetName);
} else {
reader = new InputStreamReader(bis);
}
long start = System.currentTimeMillis();
Log.info(this, "BinaryAttachmentManager getBufferredReader() reading Blob bytes into CharBuffer");
int count = reader.read(cbuf);
cbuf.limit(count); //truncate any unused space
Log.info(this, "BinaryAttachmentManager getBufferredReader() read "+count+" characters, elapsed="+(System.currentTimeMillis()-start)+" msec");


return cbuf;
------------------------------------------------------------------------------------------Out Put ----------------------------------------------------------------------------

06:34:51,190 INFO BinaryAttachmentManager,main:46 - BinaryAttachmentManager getBufferredReader() allocating 16666465 buffer with characterset null
06:34:51,331 INFO BinaryAttachmentManager,main:46 - BinaryAttachmentManager getBufferredReader() reading Blob bytes into CharBuffer
06:34:53,394 INFO BinaryAttachmentManager,main:46 - BinaryAttachmentManager getBufferredReader() read 16654336 characters, elapsed=2063 msec

----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------


 
Paul Clapham
Sheriff
Posts: 21416
33
Eclipse IDE Firefox Browser MySQL Database
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
I don't see anything in the API documentation for that read(CharBuffer) method which says that it guarantees to fill the whole buffer. Quite the contrary; it returns the number of characters that it did put into the buffer. It's your responsibility to write a loop which reads until the method returns -1, if you want to get all of the data in the reader.
 
aadhar sharma
Ranch Hand
Posts: 38
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator

Appreciate it Paul.
 
  • Post Reply
  • Bookmark Topic Watch Topic
  • New Topic