Win a copy of Cross-Platform Desktop Applications: Using Node, Electron, and NW.js this week in the JavaScript forum!
  • Post Reply Bookmark Topic Watch Topic
  • New Topic

Memory concerns in writting / appending a file  RSS feed

 
milan doshi
Ranch Hand
Posts: 43
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Hello,

We have an application where I need to put huge amount of records in a flat file. The records are text and binary (images) and the size of the file will probably run int0 700 MB to 800 MB.

I would be firing various queries/SPs and plan to append the data to the output file (The output file will be first created when the first query/SP is fired and subsequently the data will be appended to it).

My question is:

Is this a feasible idea? When I say append to an existing file, I am not sure, if that requires the ENTIRE file to be read in memory and then append the part� Is that how io works in java? If that happens than I will again run into memory issues. OR is it that when I say append to an existing file than io will only flush out my new record to an existing flat file without reading the entire file in memory(and make life simple for me ) . Kindly advice on what is the best approach for this .

Thanks in advance,
Milan
 
Jim Yingst
Wanderer
Sheriff
Posts: 18671
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
[milan]: Is this a feasible idea?

Sure.

When I say append to an existing file, I am not sure, if that requires the ENTIRE file to be read in memory and then append the part� Is that how io works in java?

No, you can append to a file without reading any of the preceding contents. FileOutputStream has a constructor that allows you to specify append mode - see the API for details. There are several other classes you could use here intead, but I think all can append easily enough. It doesn't sound like this will be a problem for you.
 
Edwin Dalorzo
Ranch Hand
Posts: 961
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
However if you need to edit a particuar record in the file and you do not want to read the whole file to find the record you want to edit, then you might consider java.io.RandomAccessFile.

But if you are just going to append it will be enough by specifying so in the constructor of your OutputStream.

You could increase performance if you use a BufferedOutputStream when writing the file to disk. Try a buffer of 1024 or 2048 bytes and see how it improves performance over an approach based on writing just one byte at a time. Because BufferedOutpuStream just writes the files to disk until the buffer is full.
 
Joe Ess
Bartender
Posts: 9425
12
Linux Mac OS X Windows
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Originally posted by Edwin Dalorzo:
Because BufferedOutpuStream just writes the files to disk until the buffer is full.


Just so we are all clear here, BufferedOutputStream writes files to an array in memory until the array is full, then it flushes the contents to disk. This can increase performance because in general, one big write is less expensive than lots of little writes. As with all performance, your results may vary. There is no substitute for benchmarking and experementing.
The default size of the buffer in BufferedOutputStream (JDK 1.5.0_03) is 8k.
 
  • Post Reply Bookmark Topic Watch Topic
  • New Topic
Boost this thread!