• Post Reply Bookmark Topic Watch Topic
  • New Topic
programming forums Java Mobile Certification Databases Caching Books Engineering Micro Controllers OS Languages Paradigms IDEs Build Tools Frameworks Application Servers Open Source This Site Careers Other all forums
this forum made possible by our volunteer staff, including ...
Marshals:
  • Campbell Ritchie
  • Bear Bibeault
  • Ron McLeod
  • Jeanne Boyarsky
  • Paul Clapham
Sheriffs:
  • Tim Cooke
  • Liutauras Vilda
  • Junilu Lacar
Saloon Keepers:
  • Tim Moores
  • Stephan van Hulst
  • Tim Holloway
  • fred rosenberger
  • salvin francis
Bartenders:
  • Piet Souris
  • Frits Walraven
  • Carey Brown

Java 8, file search and replace

 
Greenhorn
Posts: 2
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
I need to process huge text files line by line. Using Java 8 streams makes for a speedy and memory conserving solution:



Now I would like to improve speed even further. My search/replace code consists of a number of classes working on a line at a time. The order of the search/replace classes is significant and I also need to keep the order of the lines in the resulting file. This stops me from making the stream parallell. :-(

Would it make sense to use a ForkJoin solution with some kind of queue? Loading x thousands of lines into a queue. Then wrapping the replace class list in a ForkJoinTask working the queue. When work is completed append the modified queue to file.

I wonder if the overhead would be worth it?


 
Bartender
Posts: 3323
86
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
You'll probably get a better answer in the Performance forum so I'll copy it there for you.
 
Author and all-around good cowpoke
Posts: 13078
6
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Given the requirement for keeping the order of lines in a file and the order of search/replace operations in a line, the first thing that comes to mind is separate threads for reading, processing and writing lines to the new file.

This could be as simple as using java.io.PipedReader or you might maintain a queue of input lines. See also the java.util.concurrent classes.

The idea of course is that IO operations get handed over to the operating system while your search/replace methods can proceed.

Bill
 
If you want to look young and thin, hang around old, fat people. Or this tiny ad:
Thread Boost feature
https://coderanch.com/t/674455/Thread-Boost-feature
    Bookmark Topic Watch Topic
  • New Topic