• Post Reply Bookmark Topic Watch Topic
  • New Topic
programming forums Java Mobile Certification Databases Caching Books Engineering Micro Controllers OS Languages Paradigms IDEs Build Tools Frameworks Application Servers Open Source This Site Careers Other Pie Elite all forums
this forum made possible by our volunteer staff, including ...
Marshals:
  • Campbell Ritchie
  • Jeanne Boyarsky
  • Ron McLeod
  • Paul Clapham
  • Liutauras Vilda
Sheriffs:
  • paul wheaton
  • Rob Spoor
  • Devaka Cooray
Saloon Keepers:
  • Stephan van Hulst
  • Tim Holloway
  • Carey Brown
  • Frits Walraven
  • Tim Moores
Bartenders:
  • Mikalai Zaikin

memory/cpu issue of reading/writing files

 
Ranch Hand
Posts: 133
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Hi,

I am doing a project for monitoring the log files. When application(tomcat) start, it start one thread for one monitored file. The thread will run once very few seconds(pause by sleeping). Each time it will open and read the log file(bufferedReader) from the end line of previous reading. if any pre-defined keyword found in the log file, the matched line will be stored to stringbuilder, once it finish reading or the stringbuilder is bigger than a size, it will open particular file and write mathced content to the file.

The issue is that the (daily) log file will grow to be big (hundred Megebyte) from morning to the evening, and I noticed the cpu is 100% running when the file is big. I am wondering if there is good way to do this task like Bare Tail does which do not have much affect on memory and cpu?

Thanks.
 
Author and all-around good cowpoke
Posts: 13078
6
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator

Each time it will open and read the log file(bufferedReader) from the end line of previous reading.



Exactly how are you positioning at the end line of previous reading?

That is not something possible with BufferedReader which would have to read the whole file, resulting in what you observe.

Also note that a Reader is going to be doing character by character conversion to UNICODE, an expensive operation.

Bill
 
Bartender
Posts: 1558
5
Eclipse IDE Java Linux
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator

david arnold wrote:Each time it will open and read the log file(bufferedReader) from the end line of previous reading.


Instead of just opening a file after a time interval, I would suggest something like checking last modified time. If file is not modified at all, there's no point in opening the file in first place.

david arnold wrote:if any pre-defined keyword found in the log file, the matched line will be stored to stringbuilder


I'm not sure which OS you are working with, but in Linux, this can simply be done by grep command. No need to even open a file using Java classes. On Windows, there might be something similar to it.

I hope this helps.
 
david arnold
Ranch Hand
Posts: 133
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Thank you William and Anayonkar.

Exactly how are you positioning at the end line of previous reading?

That is not something possible with BufferedReader which would have to read the whole file, resulting in what you observe.



Since I used bufferedReader to read line by line, I have counter to remember the line number.

Also note that a Reader is going to be doing character by character conversion to UNICODE, an expensive operation.



Thank you. I did not know it, but I was wondering maybe it will be more efficient if I use stream. The reason i use bufferedReader is that it help to maitain the line by line format, when a line is read and it has keyword inside, it will be recorded with "\n" to the file, later it will be displayed on a browser and may need to send to email as well. so i do not know if i use stream, it will still easy to have this line format maintained.

Instead of just opening a file after a time interval, I would suggest something like checking last modified time. If file is not modified at all, there's no point in opening the file in first place.



Good point, i will add that.

I'm not sure which OS you are working with, but in Linux, this can simply be done by grep command. No need to even open a file using Java classes. On Windows, there might be something similar to it.



It is windows, it will be wonderful if it has such feature, but I am wondering if there is any.

Thanks again for the help!

Regards,
 
author
Posts: 23951
142
jQuery Eclipse IDE Firefox Browser VI Editor C++ Chrome Java Linux Windows
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator

david arnold wrote:

Exactly how are you positioning at the end line of previous reading?

That is not something possible with BufferedReader which would have to read the whole file, resulting in what you observe.



Since I used bufferedReader to read line by line, I have counter to remember the line number.




You can try the java.io.RandomAccessFile class, which can be used to seek to the point that you want, without having to read everything all over again.

Henry
 
david arnold
Ranch Hand
Posts: 133
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator

You can try the java.io.RandomAccessFile class, which can be used to seek to the point that you want, without having to read everything all over again.



Thank you Henry. I did think about using the RandomAccessFile before, i will try that.

Currently, I open the file and read and close it every few seconds, I am wondering if I should open the file, then have a loop checking for new data coming in without close it. I am not sure which way is better?

Thanks.
 
Greenhorn
Posts: 27
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
I wonder if that will work, I thought that when you opened a java.io.File for reading a snapshot of the file is opened in memory.

Edit: Maybe the line that's read is the only thing stored in memory and not the file itself.
 
Consider Paul's rocket mass heater.
reply
    Bookmark Topic Watch Topic
  • New Topic