Bookmark Topic Watch Topic
  • New Topic

Efficient File Reading  RSS feed

 
Pradeep Kalyan
Greenhorn
Posts: 6
  • Mark post as helpful
  • send pies
  • Report post to moderator
Hello

We have an application that runs on PC. It reads a input file with half million records and loads them into cache (EHCACHE), before the user work on it. Currently its taking 5 minutes to do this operation. I am just wondering if there is more efficient way to read and load into cache. (may be using streams than readers etc).

reader = new BufferedReader(new FileReader(file));
while(( line = reader.readLine() ) != null ){
fds = new FlightScheduleData( line );
lastIndex = ++lastIndex;
primaryKeyList.add( key );
CacheMgr.getCacheMain().put( key, fds);
}

Also can any help let me know, if I use multiple threads, it may improve the performance (even though the pc has only one cpu). I appreciate any input.

Thanks
Pradeep
 
Keith Lynn
Ranch Hand
Posts: 2409
  • Mark post as helpful
  • send pies
  • Report post to moderator
You might try using the read(char[],int,int) method in BufferedReader to read large arrays of chars from the file instead of reading line by line. I don't know how much that will improve performance, but you can try it.
 
Edwin Dalorzo
Ranch Hand
Posts: 961
  • Mark post as helpful
  • send pies
  • Report post to moderator
I am going to suggest something that may sound a little orthodox. You could use RandomAccessFile to read chunks of data using multiple threads.

One trhead may read from 0 upto 1 MB and have as many threads as megabytes the file have.

Using the seek method you can start reading from any position. Assign each thread a position and read from them until every thread reaches its threadshold.

Using multiple threads will guarantee that you will load the file faster.

I have never tested this idead. I supposing that if you open the file just for reading it is possible to create mutltiple RandomAccessFile object on the same file.

What do you think?

Regards,
Edwin Dalorzo.
[ April 04, 2006: Message edited by: Edwin Dalorzo ]
 
Jim Yingst
Wanderer
Sheriff
Posts: 18671
  • Mark post as helpful
  • send pies
  • Report post to moderator
Please do not post the same question in multiple forums here. It wastes people's time when they reply to a thread in one forum without realizing that the same point has already been made in another forum. Putting the discussion in one place helps people benefit from cumulative knowledge and contributions.

Since I think Performance is more appropriate as a forum choice here, I am closing this thread. Please post followups here. Thanks.
 
    Bookmark Topic Watch Topic
  • New Topic
Boost this thread!