Win a copy of Head First Android this week in the Android forum!
  • Post Reply Bookmark Topic Watch Topic
  • New Topic
programming forums Java Mobile Certification Databases Caching Books Engineering Micro Controllers OS Languages Paradigms IDEs Build Tools Frameworks Application Servers Open Source This Site Careers Other Pie Elite all forums
this forum made possible by our volunteer staff, including ...
Marshals:
  • Tim Cooke
  • Campbell Ritchie
  • Paul Clapham
  • Ron McLeod
  • Liutauras Vilda
Sheriffs:
  • Jeanne Boyarsky
  • Rob Spoor
  • Bear Bibeault
Saloon Keepers:
  • Jesse Silverman
  • Tim Moores
  • Stephan van Hulst
  • Tim Holloway
  • Carey Brown
Bartenders:
  • Piet Souris
  • Al Hobbs
  • salvin francis

Writing huge sized file data(more than 100Mb) to a output stream(converting file data to byte array)

 
Greenhorn
Posts: 20
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Hi All,

I have a file which is more than 100MB which needs to be written to a output stream (ServletOutputStream) as a response to a request where we can write byte array to a output stream. (outputStream.write(byte[])

Currently, i am using apache commons IOUtils package,

IOUtils.toByteArray(InputStream inputStream)
but this is not working when the file size is more.


Please let me know the better approach with good performance to handle huge files like more than 100MB.

Thanks,
Chinnu
 
Marshal
Posts: 26912
82
Eclipse IDE Firefox Browser MySQL Database
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Create a buffer with a reasonable size (maybe 10K bytes or 100K bytes). Write simple code which reads from the input and writes to the output, one buffer at a time. Don't write code which attempts to load the "huge" file completely into memory.
 
Chetan Ram
Greenhorn
Posts: 20
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Thanks for the reply.

Does this affect performance since every 100 Kb we need to read from input and buffer it and then write to a file every time?
Does this make process slow? Please give me a sample if possible.
 
Paul Clapham
Marshal
Posts: 26912
82
Eclipse IDE Firefox Browser MySQL Database
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
I reviewed my post and I don't see where I recommended writing to a file. What I said was

Paul Clapham wrote:Write simple code which reads from the input and writes to the output, one buffer at a time.


In your case "the output" would obviously refer to the ServletOutputStream which you referred to.

As for the "sample", it's hard to give a sample of something which is only about 5 lines of code.
 
You showed up just in time for the waffles! And this tiny ad:
Building a Better World in your Backyard by Paul Wheaton and Shawn Klassen-Koop
https://coderanch.com/wiki/718759/books/Building-World-Backyard-Paul-Wheaton
reply
    Bookmark Topic Watch Topic
  • New Topic