Bookmark Topic Watch Topic
  • New Topic
programming forums Java Mobile Certification Databases Caching Books Engineering Micro Controllers OS Languages Paradigms IDEs Build Tools Frameworks Application Servers Open Source This Site Careers Other Pie Elite all forums
this forum made possible by our volunteer staff, including ...
Marshals:
  • Campbell Ritchie
  • Tim Cooke
  • Devaka Cooray
  • Ron McLeod
  • Jeanne Boyarsky
Sheriffs:
  • Liutauras Vilda
  • paul wheaton
  • Junilu Lacar
Saloon Keepers:
  • Tim Moores
  • Stephan van Hulst
  • Piet Souris
  • Carey Brown
  • Tim Holloway
Bartenders:
  • Martijn Verburg
  • Frits Walraven
  • Himai Minh

Efficient File Reading

 
Greenhorn
Posts: 6
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Report post to moderator
Hello

We have an application that runs on PC. It reads a input file with half million records and loads them into cache (EHCACHE), before the user work on it. Currently its taking 5 minutes to do this operation. I am just wondering if there is more efficient way to read and load into cache. (may be using streams than readers etc).

reader = new BufferedReader(new FileReader(file));
while(( line = reader.readLine() ) != null ){
fds = new FlightScheduleData( line );
lastIndex = ++lastIndex;
primaryKeyList.add( key );
CacheMgr.getCacheMain().put( key, fds);
}

Also can any help let me know, if I use multiple threads, it may improve the performance (even though the pc has only one cpu). I appreciate any input.

Thanks
Pradeep
 
Ranch Hand
Posts: 2412
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Report post to moderator
You might try using the read(char[],int,int) method in BufferedReader to read large arrays of chars from the file instead of reading line by line. I don't know how much that will improve performance, but you can try it.
 
Ranch Hand
Posts: 961
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Report post to moderator
I am going to suggest something that may sound a little orthodox. You could use RandomAccessFile to read chunks of data using multiple threads.

One trhead may read from 0 upto 1 MB and have as many threads as megabytes the file have.

Using the seek method you can start reading from any position. Assign each thread a position and read from them until every thread reaches its threadshold.

Using multiple threads will guarantee that you will load the file faster.

I have never tested this idead. I supposing that if you open the file just for reading it is possible to create mutltiple RandomAccessFile object on the same file.

What do you think?

Regards,
Edwin Dalorzo.
[ April 04, 2006: Message edited by: Edwin Dalorzo ]
 
Wanderer
Posts: 18671
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Report post to moderator
Please do not post the same question in multiple forums here. It wastes people's time when they reply to a thread in one forum without realizing that the same point has already been made in another forum. Putting the discussion in one place helps people benefit from cumulative knowledge and contributions.

Since I think Performance is more appropriate as a forum choice here, I am closing this thread. Please post followups here. Thanks.
 
What do you have to say for yourself? Hmmm? Anything? And you call yourself a tiny ad.
the value of filler advertising in 2021
https://coderanch.com/t/730886/filler-advertising
    Bookmark Topic Watch Topic
  • New Topic