Win a copy of Cross-Platform Desktop Applications: Using Node, Electron, and NW.js this week in the JavaScript forum!
  • Post Reply Bookmark Topic Watch Topic
  • New Topic

Writing huge sized file data(more than 100Mb) to a output stream(converting file data to byte array)  RSS feed

 
Chetan Ram
Greenhorn
Posts: 20
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Hi All,

I have a file which is more than 100MB which needs to be written to a output stream (ServletOutputStream) as a response to a request where we can write byte array to a output stream. (outputStream.write(byte[])

Currently, i am using apache commons IOUtils package,

IOUtils.toByteArray(InputStream inputStream)
but this is not working when the file size is more.


Please let me know the better approach with good performance to handle huge files like more than 100MB.

Thanks,
Chinnu
 
Paul Clapham
Sheriff
Posts: 22374
42
Eclipse IDE Firefox Browser MySQL Database
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Create a buffer with a reasonable size (maybe 10K bytes or 100K bytes). Write simple code which reads from the input and writes to the output, one buffer at a time. Don't write code which attempts to load the "huge" file completely into memory.
 
Chetan Ram
Greenhorn
Posts: 20
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Thanks for the reply.

Does this affect performance since every 100 Kb we need to read from input and buffer it and then write to a file every time?
Does this make process slow? Please give me a sample if possible.
 
Paul Clapham
Sheriff
Posts: 22374
42
Eclipse IDE Firefox Browser MySQL Database
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
I reviewed my post and I don't see where I recommended writing to a file. What I said was
Paul Clapham wrote:Write simple code which reads from the input and writes to the output, one buffer at a time.

In your case "the output" would obviously refer to the ServletOutputStream which you referred to.

As for the "sample", it's hard to give a sample of something which is only about 5 lines of code.
 
  • Post Reply Bookmark Topic Watch Topic
  • New Topic
Boost this thread!