Win a copy of Murach's Python Programming this week in the Jython/Python forum!
  • Post Reply Bookmark Topic Watch Topic
  • New Topic

Using threads to access data from files  RSS feed

 
Pradeep Kadambar
Ranch Hand
Posts: 148
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
I am trying to use threads to access data from a list of files and directories which are passed as an ArrayList.

I am thinking of using a maximum of 10 threads. So can anyone help me with this approch.

I would as appreciate any better suggestions to make this process faster.
 
Pradeep Kadambar
Ranch Hand
Posts: 148
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Will it be ok to spawn as many threads as the number of files? And, should I synhonize the ArrayList so that no two threads read the same file?
 
Stan James
(instanceof Sidekick)
Ranch Hand
Posts: 8791
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Doing file IO in separate threads can give you a dramatic performance improvement up to a point. When you get so many threads going that the disk head is banging back and forth too much or you've saturated the CPU handling the data the benefits go away and might start hurting you. So, no, don't blindly create a thread per file. It might hurt you.

As far as synchronizing the list, just don't let the threads have access to the list.

Now back to how many threads ... that code would start a thread per file, which would probably be bad. Look into Java 5.0s Executor and other thread pool goodies. If you're pre 5.0 look at Jakarta Commons Thread Pool. They both give you a cool way to limit the number of threads running at one time.

Hope that helps!
 
Pradeep Kadambar
Ranch Hand
Posts: 148
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Can you just show me how to use threads to read data from files and directories whose names are passed in an arraylist. (As I am not well versed in threads)
 
Stan James
(instanceof Sidekick)
Ranch Hand
Posts: 8791
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
The example above has the heart of it. Iterate through the Files or filenames in the list. For each one, create a new instance of a class that implements Runnable - I called it FileHandler - and start it up on a new thread. Do the reading in the run() method of that object. I used some Java 5.0 syntax there; if it's not familiar just do the normal Iterator and hasNext() kind of thing.

You'll probably want to collect the results from all that reading. You might pass each FileHandler a reference to a DataCollector, and have the FileHandler call dataCollector.putResults().

This is not compiled or tested, may have lots of errors ...

Does that help?

Geeze, did I just do somebody's homework?
[ October 26, 2004: Message edited by: Stan James ]
 
  • Post Reply Bookmark Topic Watch Topic
  • New Topic
Boost this thread!