• Post Reply
  • Bookmark Topic Watch Topic
  • New Topic

File is open for long

 
Matron disney
Greenhorn
Posts: 7
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
I have an application which opens a file does processing.The file is a big one so we do this in a loop.We read every 1000 lines once ina loop.the whole reading takes approx 20 sec but processing takes 2-3 hours as there are hughe processing logic for that stream.after the (reading+processing) we close the stream.Is this a problem.Will this be consuming more resources as compared to closing the stream in every loop.
Has anybody got any problems not closing the stream for so long.Mind you this is an input stream and not an output stream.
 
David Harkness
Ranch Hand
Posts: 1646
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
I see no problem with keeping an InputStream open for any length of time. System.in (the keyboard or redirected input, normally) is open for the entire run of your program.
 
Matron disney
Greenhorn
Posts: 7
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Thanks for the reply.Just one correction.We do have many files let us say 100 in no.so as the processing for all thesfiles takes couple of hours and reading just a minute or two don't you think it is better to close the file after every chunk of reading.The performance may hit but at least we will not 50 readers open at the same time.My question is;
1)Is reader is associated with system resource which by closing will release it.
2)Is there any limit of opening the readers at one time.
 
Joe Ess
Bartender
Posts: 9313
10
Linux Mac OS X Windows
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Originally posted by Matron disney:

1)Is reader is associated with system resource which by closing will release it.

Yes.

Originally posted by Matron disney:

2)Is there any limit of opening the readers at one time.

Yes. It is not uncommon to get an error which indicates that there are too many files open with poorly tuned database applications.
 
Stan James
(instanceof Sidekick)
Ranch Hand
Posts: 8791
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
This is taking me back to mainframe days. Do you have to worry about checkpoints and restart? If the thing crashes halfway through would it be good to know the first n records or files processed fine and you can restart at n+1 next time? That might influence how much you read at a time and how you treat the file between reads.
 
Matron disney
Greenhorn
Posts: 7
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
No that is not a big concern.The concern which we have now is should we open the file read the file in chunks,do the processing ,again read in chunks do the processing till athe whole file is processed and then close the reader/file or
we read the file in chunks ,do the processing close the file again open the file and so on.
The processing time in most of these is 1-2 hours,reading is 3-4 min.
 
Henry Wong
author
Marshal
Pie
Posts: 21430
84
C++ Chrome Eclipse IDE Firefox Browser Java jQuery Linux VI Editor Windows
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Originally posted by Matron disney:
No that is not a big concern.The concern which we have now is should we open the file read the file in chunks,do the processing ,again read in chunks do the processing till athe whole file is processed and then close the reader/file or
we read the file in chunks ,do the processing close the file again open the file and so on.
The processing time in most of these is 1-2 hours,reading is 3-4 min.


Given modern day JVMs and operating systems, I don't think is a problem keep the files open... However, having a device driver development background, I cringe everytime someone abuses resources, when they don't have to. I say if it not a concern, then close those files.

Henry
[ April 28, 2005: Message edited by: Henry Wong ]
 
Matron disney
Greenhorn
Posts: 7
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
why type of resources are consumed as I didn't see any increase in memory consumption when more readers are involved.
 
  • Post Reply
  • Bookmark Topic Watch Topic
  • New Topic