• Post Reply Bookmark Topic Watch Topic
  • New Topic

reading from a file in real time.  RSS feed

 
Tejaswini Shirkhedkar
Greenhorn
Posts: 17
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
I need to read from a huge file and analyse the data and then write it in the excel file.
The process should be real time. i.e. when the data is available, the program should start reading it and when it is not, program should wait till it gets available.
What java API should I refer to?
Please respond.

Tejaswini
 
Joseph Maddison
Ranch Hand
Posts: 53
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
What you're describing isn't quite realtime per se, but it may have scared off folks from answering your question. If I understand correctly, you want to have some process create a file, and when it's done, you'd like your Java application to read the file, convert it to Excel, and then delete the file... and wait to do it again.

What you don't want, presumably, is for your Java application to read the file while it is still being written, nor do you want the other application to write a new file while the old version is being read.

There may be plenty of clever ways to do this, but keeping to the philosophy of "keep it simple", when I did something like this, I used file access to determine if a process could consider itself the file owner for a brief period of time. To keep that ownership, the application renames the file to a different name of its own choice. So...

  • application #1 writes huge data file "x", deleting the old "x" if it's there first, then repeats on some unknown schedule.
  • application #2 looks for file "x". This could be a loop, say sleeping 10 seconds and then checking if "x" exists and then checking to see if it can successfully rename "x" to "y" if it's there. If it can't, #1 is still writing the file, and so #2 simply sleeps a little longer and tries again.
  • application #2 processes "y" and writes it to Excel file "z". This can be concurrent with #1 writing another "x"-- it doesn't matter, since they are separate files.
  • application #2 deletes "y" and starts over, looking for "x" as above.


  • Hope this helps,
    jdmaddison
     
    Tejaswini Shirkhedkar
    Greenhorn
    Posts: 17
    • Mark post as helpful
    • send pies
    • Quote
    • Report post to moderator
    Thanks for replying, I had almost lost any hope..

    Well, to describe a little more what I want to do..
    I want to write a Java program that reads from the logfile while it is being generated. What I understand from the logfile is, records of website traffic are added to the logfile as the website receives hits..

    The website in question doe not receive any hits after end of the day.
    I have written a program that analyses the logs at the end of the day i.e. when all the transacations are over and there are no new additions to the file, the program will take that file and process the data.
    Now I want to modify it to make it real time.

    i.e. instead of processing the logfile at the end of the day, I want my program to take a line from the log file as soon as it is added to it, process it, wait till next line is available, grab it as soon as it becomes available and so on till we reach the end of the day.

    I have never done anything so complicated before, have been searching the net for quite a few days now but could not find any idea suitable to my needs.

    I will be really grateful if I get any help in accomplishing this.
    Thanks in advance.
    Looking forward for inputs.

    Tejaswini
     
    Tejaswini Shirkhedkar
    Greenhorn
    Posts: 17
    • Mark post as helpful
    • send pies
    • Quote
    • Report post to moderator
    Please help!!!
     
    David Harkness
    Ranch Hand
    Posts: 1646
    • Mark post as helpful
    • send pies
    • Quote
    • Report post to moderator
    What you're describing is far simpler than you're making it sound. I'll assume you're sticking with the standard stream-based Java classes (in other words, you're not using JDK 1.4's New Input/Output classes). If that's not the case, the same applies but the API is different.

    Since the weblogs are line-oriented, let's keep it simple to start.Note that done is never set to true. Hit control-C to kill the process or put the above into a Thread and have the main Thread create a basic UI to allow you to stop it gracefully.

    If the end of the file is reached, instead of stopping the program sleeps (idles) for 1 second, which you can change to a larger or smaller value to affect its responsiveness. When it wakes up, it reads again. It keeps doing this until the web server writes to the file (and flushes its buffer to disk), at which point this program reads the new lines and keeps going.

    Hope this helps some.
     
    Tejaswini Shirkhedkar
    Greenhorn
    Posts: 17
    • Mark post as helpful
    • send pies
    • Quote
    • Report post to moderator
    Thanks!!
    This is my first job and so its difficult for me to start writing real world programs right away.
    I know I should know more..

    Teju
     
    Joseph Maddison
    Ranch Hand
    Posts: 53
    • Mark post as helpful
    • send pies
    • Quote
    • Report post to moderator
    Once you get it working with a sequential File Reader, you might look into using the RandomAccessFile handling to continue reading from where you left off (without having to re-read the previous lines). You will need to store the last position (that was end of file on the previous repetition), close the old Stream, then open a RandomAccessFile and seek to the last position.

    Hope this helps,
    jdmaddison
     
    David Harkness
    Ranch Hand
    Posts: 1646
    • Mark post as helpful
    • send pies
    • Quote
    • Report post to moderator
    Originally posted by Joseph Maddison:
    Once you get it working with a sequential File Reader, you might look into using the RandomAccessFile handling to continue reading from where you left off (without having to re-read the previous lines).
    Unless I'm missing something, as long as the File is kept open this won't be necessary. However, using the RandomAccessFile and saving the last position would allow the process to be stopped (and the File closed) and restarted later.
     
    • Post Reply Bookmark Topic Watch Topic
    • New Topic
    Boost this thread!