You will have to repeatedly open the file just long enough to read from it, then close it as soon as possible - you don't want your reads interfering with possible write. You will have to monitor the folder to look for roll-over files. When you read from a file you won't know if you reached the end of the file because you caught up to the writes, or if you reach the end of the file because the log file rolled over to another one. So when you read the EOF,
you should check for the existence of the next file - if it doesn't exist then try re-reading the current one.
When you read from the file give up time to allow the writing task to gain access to the file. Use only one
Thread to actually access the file - unless you have a multi-headed disk. Reading from the same disk in multiple threads usually just slows things down. Once you read from the file you can, of course, distribute the work required to multiple threads...