• Post Reply Bookmark Topic Watch Topic
  • New Topic

Lots of .nfs files being created  RSS feed

 
Andrew Robinson
Greenhorn
Posts: 15
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
I wrote a program that needs to parse through close to 4MB of data that is a compilation of smaller blocks of data which in turn are compilations of even smaller messages. I currently read a block from the file and then write that block to a temp file in order to read/parse through the smaller messages. I noticed that each time I create and then delete the temp file a .nfs[some hex numbers] is created and it creates close to 6000 of these files. I was wondering if there was any way to avoid this because it takes 1-2 minutes to parse a 4 MB file.
 
Greg Charles
Sheriff
Posts: 3014
12
Firefox Browser IntelliJ IDE Java Mac Ruby
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Probably you are running Unix or some derivative, and you are creating the temporary files on some remote file system. I could go into why .nfs files get created and what causes them to linger, but it's probably simpler for you to just create your temporary files on the local file system or, even better, don't create temporary files at all ... just parse your smaller messages in memory and then move on.
 
Andrew Robinson
Greenhorn
Posts: 15
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
yes they are on a remote file system. Its the easist way I could think to parse the data. Do you think that the creation of all of those files is what's causing it to run so slow and if I parsed it in memory it would be much quicker??
 
Greg Charles
Sheriff
Posts: 3014
12
Firefox Browser IntelliJ IDE Java Mac Ruby
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Yes, doing things in memory is always faster assuming you have the memory to spare. Disk I/O and network I/O are much slower ... imagine a tortoise racing with a comet. There can be legitimate reasons for using temp files, but definitely avoid them if you can.
 
Andrew Robinson
Greenhorn
Posts: 15
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
The data is a bunch of binary data so shoud I just make a massive byte[] and copy to that rather than writing to a temp file???
 
Greg Charles
Sheriff
Posts: 3014
12
Firefox Browser IntelliJ IDE Java Mac Ruby
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Well, 4 Mb isn't really massive, so bringing it all into memory should be fine. However, you were splitting it up into smaller temporary files before. Wouldn't it be possible to read one chunk of the file, do whatever you need to do with it, and then read the next chunk? Do you need the whole file in memory at once?

I once rewrote a process that was reading a large XML file from a database record, transforming it with a DOM parser, then zipping the result, adding a header, and sending it to a remote client. The way it was implemented, it actually kept six copies of the file in memory in various forms, so as the files started getting larger, the thing became a huge memory hog. Through judicious use of IO streams (and a change to a SAX parser), I was able to read the file, transform, and send it with never more than a single buffer-full of it in memory at any given time.
  • Post Reply Bookmark Topic Watch Topic
  • New Topic
Boost this thread!