• Post Reply Bookmark Topic Watch Topic
  • New Topic
programming forums Java Mobile Certification Databases Caching Books Engineering Micro Controllers OS Languages Paradigms IDEs Build Tools Frameworks Application Servers Open Source This Site Careers Other Pie Elite all forums
this forum made possible by our volunteer staff, including ...
Marshals:
  • Campbell Ritchie
  • Jeanne Boyarsky
  • Ron McLeod
  • Paul Clapham
  • Liutauras Vilda
Sheriffs:
  • paul wheaton
  • Rob Spoor
  • Devaka Cooray
Saloon Keepers:
  • Stephan van Hulst
  • Tim Holloway
  • Carey Brown
  • Frits Walraven
  • Tim Moores
Bartenders:
  • Mikalai Zaikin

Is there any efficient and tested java util library that has a java version of the tail command?

 
Ranch Hand
Posts: 53
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Hi,

I'm wondering if anyone knows of any java util library (like Apache Commons) that has an efficient and tested java implementation of the unix tail command? By effective I mean that it should be able to handle very large files (>1GB), with various unknown line lengths, quickly without lots of I/O or high memory usage. And by tested I mean something that can safely be used in production environments. In short, something as safe and efficient (well almost at least) as the tail command on some unix OS.

The reason I'm asking this is because I want to be able to display a server log file in a administration webapplication, and since the log file can grow very large I only want to show the last N lines. But I was amazed to notice that I couldn't seem to find any "java tail" function. I found a lot of different code examples, but that doesn't help me much. This seems like such a low level function that it should really exist in a tested, efficient and hopefully nearly bug free java util library. Writing it myself (even if that means copy pasting example code from some forum) is way to error prone for my taste.

So, does anyone know of any such existing such java util library? Preferably one that exists in a standard maven repository.

Regards
/Jimi
 
Rancher
Posts: 43081
77
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
If it's a web app, what part do you envision Java to play? Streaming log entries to an HTML/JavaScript client? Or in an applet that reads from a streamed server source?
 
Jimi Svedenholm
Ranch Hand
Posts: 53
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator

Ulf Dittmer wrote:If it's a web app, what part do you envision Java to play? Streaming log entries to an HTML/JavaScript client? Or in an applet that reads from a streamed server source?



It would simply be a page displaying the last say 1000 lines of the file, in a scrollable div, that would do an ajax refresh automatically every 30 seconds or so (with the possibility to turn that off using a check box on the side) and scroll the div to the end on each update.
 
Ranch Hand
Posts: 781
Netbeans IDE Ubuntu Java
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
It may not be tested to the level you require but reply #10 of http://forums.sun.com/thread.jspa?threadID=709737.

Of course you could always write a test harness for yourself!
 
Jimi Svedenholm
Ranch Hand
Posts: 53
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator

James Sabre wrote:It may not be tested to the level you require but reply #10 of http://forums.sun.com/thread.jspa?threadID=709737.

Of course you could always write a test harness for yourself!



Well... As I said, it is very easy to copy paste some example code from the net. But that is not what I am looking for. How do I know that anyone else is using this code currently? And can I get news of any bugs found? The most likely scenario is that if anyone uses that code and then later finds some bugs he simply tries to fix the bugs himself and I have no idea about it.

But the lack of response seems to indicate that what I am looking for doesn't exist. And I guess that the example code you link to is better then nothing. Although I just tested it, and it can't handle windows line breaks (CR + LF becomes LF + CR and that is interpreted as two separate line breaks), so I had to fix that. Also it doesn't handle UTF-8, but that I can live with I guess.

Thanks for the tip, even though I'm still a bit perplexed about the fact that I have to reinvent the wheel like this.

Regards
/Jimi
 
Marshal
Posts: 28193
95
Eclipse IDE Firefox Browser MySQL Database
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Going back one step from your "tail" requirement to your original requirement of being able to display log entries from one application in a second application...

If I wanted to do that, here's what I would do:

  • Use log4j in the application doing the logging
  • Configure it to use a SocketAppender
  • Have the second application run a SocketServer to receive the logs


  • Now perhaps you can't do that, in which case just disregard the advice. But I have actually done that (in my case the second application actually needed to act on entries in the first application's logs.)
     
    James Sabre
    Ranch Hand
    Posts: 781
    Netbeans IDE Ubuntu Java
    • Mark post as helpful
    • send pies
      Number of slices to send:
      Optional 'thank-you' note:
    • Quote
    • Report post to moderator


    Well... As I said, it is very easy to copy paste some example code from the net. But that is not what I am looking for.


    Which is why I was slow pointing you at it.


    How do I know that anyone else is using this code currently?


    You don't but just because someone may be using it does not make it perfect. I needed an ASCII85 encoder for a project and quickly turned up one from FREEHEP. It was rubbish. I created a simple JUnit test harness which quickly found 3 show stopper bugs. I reported the bugs to FREEHEP and a long time later I got some feedback saying that they had fixed at least two of the bugs but I have lost confidence in the whole FREEHEP site. While they were fixing a couple of bugs I wrote one together with a comprehensive JUnit tests. I have confidence in my code.

    The moral of the story is - test for yourself.


    And can I get news of any bugs found?


    Most probably not unless someone posts to that thread. You can set yourself up to receive an email if anyone posts to the thread but ...


    The most likely scenario is that if anyone uses that code and then later finds some bugs he simply tries to fix the bugs himself and I have no idea about it.


    Of course.


    But the lack of response seems to indicate that what I am looking for doesn't exist. And I guess that the example code you link to is better then nothing. Although I just tested it, and it can't handle windows line breaks (CR + LF becomes LF + CR and that is interpreted as two separate line breaks), so I had to fix that. Also it doesn't handle UTF-8, but that I can live with I guess.


    I had to produce that code in the first place because, like you, I needed tail() type functionality for a home project for looking at log files. I was primarily using iso-8859-1 encoding and I think I said in the thread that it did not work for multi-byte character sets. I have moved on since then. I developed an improved version for a customer that handles the Java mandated character sets of US-ASCII, ISO-8859-1, UTF-8, UTF-16BE, UTF-16LE and UTF-16. It actually handles all the iso-8859-x character sets and may handle others by accident. The only character sets it is known to fail to handle is EBCIDIC family though there may be others. It deals with the CR-LF problem.

    The new improved version is quicker than the original and the heart of the code is actually much smaller than the original. I cannot release it since it is owned by my customer. One day I might get round to creating a more general version and release that but this won't satisfy your requirement for a significant user base and well defined bug reporting system.


    Thanks for the tip, even though I'm still a bit perplexed about the fact that I have to reinvent the wheel like this

    .
    Why perplexed? Surely this sort of deficiency is what drives the Jakarta projects.
     
    Jimi Svedenholm
    Ranch Hand
    Posts: 53
    • Mark post as helpful
    • send pies
      Number of slices to send:
      Optional 'thank-you' note:
    • Quote
    • Report post to moderator

    Paul Clapham wrote:Going back one step from your "tail" requirement to your original requirement of being able to display log entries from one application in a second application...

    If I wanted to do that, here's what I would do:

  • Use log4j in the application doing the logging
  • Configure it to use a SocketAppender
  • Have the second application run a SocketServer to receive the logs


  • Now perhaps you can't do that, in which case just disregard the advice. But I have actually done that (in my case the second application actually needed to act on entries in the first application's logs.)



    I actually had something like this in mind before, using some log4j appender to get access to the information, but when I asked in the Log4j user mailing list I only got one reply and involved a JdbcAppender. But I'm not very keen on causing more database traffic, so that is not an alternative for me. A SocketAppender sounds quite interesting, but in my case even a simple "InMemoryAppender" is enough, since the logging happens in the same webapplication.

    I actually got inspired by your suggestion, and decided to try and write my own appender, and it was quite simple. Sure, it doesn't give me access to events logged before a restart/crash, but that is a minor issue and I now provide a download link so the user can access the complete log file if needed.
     
    Sheriff
    Posts: 22783
    131
    Eclipse IDE Spring VI Editor Chrome Java Windows
    • Mark post as helpful
    • send pies
      Number of slices to send:
      Optional 'thank-you' note:
    • Quote
    • Report post to moderator

    Jimi Svedenholm wrote:I actually got inspired by your suggestion, and decided to try and write my own appender, and it was quite simple.


    Nice, isn't it?

    Sure, it doesn't give me access to events logged before a restart/crash, but that is a minor issue


    You could try parsing the existing file when the appender is created for an initial list of events. Afterwards you simply toss those away as your list gets more and more events.
     
    Jimi Svedenholm
    Ranch Hand
    Posts: 53
    • Mark post as helpful
    • send pies
      Number of slices to send:
      Optional 'thank-you' note:
    • Quote
    • Report post to moderator

    James Sabre wrote:
    I had to produce that code in the first place because, like you, I needed tail() type functionality for a home project for looking at log files. I was primarily using iso-8859-1 encoding and I think I said in the thread that it did not work for multi-byte character sets. I have moved on since then. I developed an improved version for a customer that handles the Java mandated character sets of US-ASCII, ISO-8859-1, UTF-8, UTF-16BE, UTF-16LE and UTF-16. It actually handles all the iso-8859-x character sets and may handle others by accident. The only character sets it is known to fail to handle is EBCIDIC family though there may be others. It deals with the CR-LF problem.

    The new improved version is quicker than the original and the heart of the code is actually much smaller than the original. I cannot release it since it is owned by my customer.



    ah, too bad. Code like that really belongs in a apache commons project or similar.


    Why perplexed? Surely this sort of deficiency is what drives the Jakarta projects.



    Well, yes, but I really though that this deficiency had been noticed enough times before that it already existed in such a project, just that I hadn't found it.
     
    Jimi Svedenholm
    Ranch Hand
    Posts: 53
    • Mark post as helpful
    • send pies
      Number of slices to send:
      Optional 'thank-you' note:
    • Quote
    • Report post to moderator

    Rob Prime wrote:

    Jimi Svedenholm wrote:
    Sure, it doesn't give me access to events logged before a restart/crash, but that is a minor issue


    You could try parsing the existing file when the appender is created for an initial list of events. Afterwards you simply toss those away as your list gets more and more events.



    Yes, that actually sounds like a very nice compromise. Why didn't I think of that?

    The only thing is that I noticed a problem with log4j on windows. If I read the from the file at the same time as the DailyRollingFileAppender decides its time to rename the file there can be problems. Because, on windows a file rename fails if another process/thread has the file open. And apparently Log4j doesn't handle that properly, because when that happend to me yesterday, on my development machine, log4j decided to erase everything logged the previous day, and that is totally unacceptable if you ask me. It should really be able to handle that situation (maybe by trying to rename the file again, a few seconds later, but as a last resort at least keep the old log lines intact and just keep appending to the file).

    I guess I could do a OS check, and only read from the file on non-windows machines. But now it's midsummer weekend, so no more work for mefor a few days.

    Thanks for your suggestions, Rob.
    reply
      Bookmark Topic Watch Topic
    • New Topic