• Post Reply Bookmark Topic Watch Topic
  • New Topic

Monitoring a directory?  RSS feed

 
Andrew Mcmurray
Ranch Hand
Posts: 188
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Hi all

I have to process files that are being ftped to a certain directory. I have it to where if I give it the file or directory it can process all the files, but I need a way to know when new files have arrived so I can start processing them. Do you guys know some good techniques used to moniter a certain directory?

Thanks,

AMD
 
Stan James
(instanceof Sidekick)
Ranch Hand
Posts: 8791
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
I think your two choices are to periodically list the directory and see if anything new showed up or to use JNDI to a C/C++ module using some operating system notification API. Guess the choice depends on how quickly you need to spot new files, how much overhead you can stand for listing the directory, how good your C/C++ skills are, and which operating systems you have to support. Whew. Does that sound fun or what?
 
Andrew Mcmurray
Ranch Hand
Posts: 188
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Ouch that does not sound like fun!
 
John Kelty
Ranch Hand
Posts: 33
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
You'll have to be careful to not grab any files that are still in transit and are only partially FTP'd over.
 
Andrew Mcmurray
Ranch Hand
Posts: 188
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Yes, when ftp is finished putting the file, the file is renamed so I know when it is safe to process them.
 
Ulf Dittmer
Rancher
Posts: 42972
73
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Or you could write it in Java, and have a thread periodically check which files are there. Once a new file is spotted, you wait a certain amount of time until its size no longer changes, and then operate on it. Since the ftp process will have any file open for writing that is still being uploaded, you could check whether you can also open it for writing (which would fail until the ftp process is done with it). That way you can be reasonably safe that you're not grabbing half-uploaded files.
 
Andrew Mcmurray
Ranch Hand
Posts: 188
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
I was looking around alittle is there any way to use a Listner instead of some of the above ways listed?
 
Ulf Dittmer
Rancher
Posts: 42972
73
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
If you're free in your choice of FTP server, have a look at Apache FTP server. It's written in Java, and has a concept on ftplets, kind of like servlets for HTTP, except that they are called whenever certain FTP events happen (like connect, file upload, file download, disconnect, etc.).
 
Andrew Mcmurray
Ranch Hand
Posts: 188
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Na no luck on type of ftp server, it could be coming in on all different kinds.
 
John Kelty
Ranch Hand
Posts: 33
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Originally posted by Ulf Dittmer:
you could check whether you can also open it for writing (which would fail until the ftp process is done with it). That way you can be reasonably safe that you're not grabbing half-uploaded files.


Actually, that won't work on UNIX platforms, because UNIX would let you open the file even while it is being FTP'd. I tested on Windows and UNIX with the following code. Windows gave the expected exception and UNIX let the IO operations succeed even during FTP.
 
Paul Clapham
Sheriff
Posts: 22832
43
Eclipse IDE Firefox Browser MySQL Database
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Originally posted by Ulf Dittmer:
Or you could write it in Java, and have a thread periodically check which files are there. Once a new file is spotted, you wait a certain amount of time until its size no longer changes, and then operate on it.
I have found this method to be unreliable. The renaming business that Andrew is using is what we're using with as many partners as possible, and it's about as reliable as you can get.

But I've always wondered why FTP servers don't have hooks where you can get callbacks after an upload is complete. Looks like Apache has tried to answer that question after only 40 years since FTP was invented.
[ August 17, 2006: Message edited by: Paul Clapham ]
 
Stan James
(instanceof Sidekick)
Ranch Hand
Posts: 8791
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Has anybody used stdin and stdout to interact with another process instead of JNDI? I once used a language called EASEL and that was their primary integration technology. For example we wrote SQL queries to stdin of another process and it sent CSV results back through its stdout one row at a time. Sounds clunky but it's a pretty universal lowest common denominator. Kind of like XML.
 
Paul Clapham
Sheriff
Posts: 22832
43
Eclipse IDE Firefox Browser MySQL Database
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Well, that's a pretty common technique in Unix. You take a bunch of little programs that read from stdin and write to stdout, and put | them | together | on | the | command | line | like | this.
 
Stan James
(instanceof Sidekick)
Ranch Hand
Posts: 8791
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Yes, I've used pipes between processes many times, but EASEL did it in a more synchronous, interactive mode. I made a little Java toy program that did it but was buggier than I liked.
 
Don't get me started about those stupid light bulbs.
  • Post Reply Bookmark Topic Watch Topic
  • New Topic
Boost this thread!