• Post Reply Bookmark Topic Watch Topic
  • New Topic

best way to write large data to file..  RSS feed

 
satyendra adhikari
Ranch Hand
Posts: 52
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Hi,
I have to write large volume of data fetched from database to a file. My data is initially is a two dimensional array as returned from data access classes. I need to generate a csv file. So I am going through the array and using StringBuffer to create the data that is written to file typically by FileOutputStream. (StringBuffer.getBytes())
I am not sure if creating large StringBuffer typically for thousands of records could be a problem or is a good thing. Can someone offer some advice on that or some better ways to implement this thing?
 
Joe Ess
Bartender
Posts: 9406
12
Linux Mac OS X Windows
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Well, the most important question is "does it work?". That may be Good Enough, depending on your exact requirements. If you are looking for the Most Efficient Method, there will be some balance between the amount of data you write to the string buffer before you write the data to file and the number of file accesses you will want to make. The amount of data you write to a string buffer is a limit because string buffers occupy memory and memory is finite. However adding data to a string buffer is very fast compared to writing to disk, and writing a big chunk of data to disk is faster than writing lots of small chunks. The exact place to strike this balance varies from program to program and there's only one way to find it: Benchmark your code, try something else and benchmark again. Java Platform Performance is an online book from the source, Sun, which describes many useful performance techniques as well as the methods to test them.
 
satyendra adhikari
Ranch Hand
Posts: 52
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
It works fine as of now for few hundred records but I am yet to test it against thousands of records that it is suppose to handle in main environment. I was basically curious to know if there is any optimized limit that java suggests on data kept in a StringBuffer so that I could empty it by writing to file whenever my StringBuffer reaches that capacity. Right now my code attempts to write to the file only once when all data has been stored in a StringBuffer.
 
Richard Rodger
Greenhorn
Posts: 17
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Why not just write directly to the file using a PrintWriter?
 
  • Post Reply Bookmark Topic Watch Topic
  • New Topic
Boost this thread!