• Post Reply Bookmark Topic Watch Topic
  • New Topic
programming forums Java Mobile Certification Databases Caching Books Engineering Micro Controllers OS Languages Paradigms IDEs Build Tools Frameworks Application Servers Open Source This Site Careers Other Pie Elite all forums
this forum made possible by our volunteer staff, including ...
Marshals:
  • Campbell Ritchie
  • Jeanne Boyarsky
  • Ron McLeod
  • Paul Clapham
  • Liutauras Vilda
Sheriffs:
  • paul wheaton
  • Rob Spoor
  • Devaka Cooray
Saloon Keepers:
  • Stephan van Hulst
  • Tim Holloway
  • Carey Brown
  • Frits Walraven
  • Tim Moores
Bartenders:
  • Mikalai Zaikin

Can I read a file chunks in parallel?

 
Ranch Hand
Posts: 1873
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Hi all,

After long time..

I have huge file (about 500k fixed size records or more). I wanted to know if I can read it in parallel via different threads.

I have not used Java NIO packages yet! (sounds strange, but it is true). Is there a way we can read the file in parallel using any of the NIO concepts? I am still going through tutorial and may be some experiments but I thought of asking here.

Also, if you know of any other/better way of reading a file in parallel other than NIO, it would be great if you share it here.

Regards,
Maulin
 
Rancher
Posts: 5008
38
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Only if the chunks of the file are on different drives. The OS is going to be the limiting factor on your being able to read different parts of the same file in parallel. In fact trying to do that may slow down the process because the read head on the drive could have to jump back and forth slowing down the process. If you could do the reads at a lower level (ie in machine language) and you knew the relative speed of the disk spinning vs the read time to read a sector from the disk (assuming the disk is sectored), then you could time your reads to start a read a selected sector just before the read head gets to that sector. That might make it possible to read a full track in two rotations (reading sectors 0, 2, 4 and then 1, 3 5) versus reading one sector per rotation. 6 sectors = 6 rotations vs 2.
 
Author and all-around good cowpoke
Posts: 13078
6
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Consider using one Thread to read in records and a separate Thread to process them. You might create a Queue of records waiting to be processed.

I bet you will find that the processing Thread is the limiting factor, not the reading Thread.

Bill
 
With a little knowledge, a cast iron skillet is non-stick and lasts a lifetime.
reply
    Bookmark Topic Watch Topic
  • New Topic