Win a copy of Murach's Python Programming this week in the Jython/Python forum!
  • Post Reply Bookmark Topic Watch Topic
  • New Topic

File Processing  RSS feed

 
Sri Rangan
Ranch Hand
Posts: 160
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
I have to process huge files (upto 80MB) and the structure of the file is as below. Basically a huge file has 'N' segments of data with the header and footer and may have to trim some elements at the end on each line. Then I have to take each segment (between start to end) and call my parser to create objects out of it. This is part of a high volume batch process where I will be reciveing large volume of these files.

I understand it is not advisable to read the entire file into memory which will lead to 'out of memory exception'.
I am planning to use the following approach to read the stream in chunks.



This seems to work for huge files. My question is how could I get each segment of data (between start and end) in this process so that I can take each segment and call my parser to covert objects
Thanks
Sri
 
Stefan Wagner
Ranch Hand
Posts: 1923
Linux Postgres Database Scala
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Use a BufferedReader, and read lines.
Read until line.equals ("<END>"); and store the lines in an ArrayList.
Pass the AL to a method, which parses the lines.
Repeat till eof.
[ October 07, 2004: Message edited by: Stefan Wagner ]
  • Post Reply Bookmark Topic Watch Topic
  • New Topic
Boost this thread!