Win a copy of Cross-Platform Desktop Applications: Using Node, Electron, and NW.js this week in the JavaScript forum!
  • Post Reply Bookmark Topic Watch Topic
  • New Topic

Constans access to the latest contents in a folder  RSS feed

 
alberto gonzalez
Greenhorn
Posts: 14
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Hi All,

I need to write a piece of code that will scan a folder constantly for new files. The code has to be running either on an infinite loop or use a listener so that when a new file is created in the folder a process gets triggered and the file is processed.

The things that this code has to support are:

- Real time response. A file needs to be processed as soon as the file reaches the folder (and then it will be deleted)
- The code needs to be running 24x7 365 days a year. It has to be as efficient as possible
- It would be a plus if you could have more than one process running at a time so that for those times of the day where the folder is receiving lots of files may be you could run more than one process at a time.

Has anybody done anything like this before or used any open source software to deal with a situation like this?

This is a critical part of the application and it needs to be very very stable since the application gets fed all the information through this bit of code.

I have been looking at the File class of the java.io package. Is there anyway to refresh the folder once you have the file(folder) object? Is it a good practice just to get the same folder again but now it will only contain the new files? The only problem that I can see doing this is that if the folder has no files it will enter an infinite loop of getting the folder when there is no need for it, hence why I mentioned the listerner.

Thanks for your help
 
Ernest Friedman-Hill
author and iconoclast
Sheriff
Posts: 24217
38
Chrome Eclipse IDE Mac OS X
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Hi,

Welcome to JavaRanch!

There's no mechanism you can use to be notified about new files in Java, so the best you can do is call File.listFiles() periodically. Typically, you use the Thread.sleep() call to pause a bit between polls; as long as you do that, the empty directory issue won't be much of a problem.
 
alberto gonzalez
Greenhorn
Posts: 14
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Thanks for the quick response!!

I was hoping it will be different
 
Stan James
(instanceof Sidekick)
Ranch Hand
Posts: 8791
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
A significantly more advanced option would be to write a native C/C++ module using Windows or Unix APIs to monitor a directory, and communicate with Java via JNI. If you are building a real-time user-interface driven thing like Eclipse this could give a more professional user experience.

If you're building a server that picks up new files people drop in a directory it might be good enough to check every 10 seconds or every minute with the other technique. Surely it would be a lot simpler!
 
alberto gonzalez
Greenhorn
Posts: 14
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Thanks for your help,

The application is a server side one. Sometimes you get as much as 10-15 files per second entering the folder and sometimes none for a few seconds but when they enter they need to be processed and available to the user as soon as possible. I am thinking that I will probably need to use an infinite loop. Do you still think that the solution proposed will be good enough?

Thanks again
 
Jim Yingst
Wanderer
Sheriff
Posts: 18671
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Yeah, probably. It's a bit ugly, but an infinite loop works pretty well here. I'd allow for some configurable delay time between checks. From what you've said, I might start with 100 milliseconds. Ought to be fast enough, while still being enough delay that the processor doesn't just spend all its time executing the loop. You can study the performance stats of your system while it's operating, and adjust the delay accordingly. Note that you can probably save yourself a bit of coding here by using java.util.Timer.

[alberto]: I have been looking at the File class of the java.io package. Is there anyway to refresh the folder once you have the file(folder) object? Is it a good practice just to get the same folder again but now it will only contain the new files?

You're always checking the same directory, right? There should be no need to create a new File object for that directory - reusing the object is fine. Every time you call listFiles(), you will get a fresh view of the files in that directory at the time.
 
alberto gonzalez
Greenhorn
Posts: 14
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Thanks a lot,

I did look for a refreshFolder() method but it doesn't exist. I'll try your approach.

Thanks again
 
alberto gonzalez
Greenhorn
Posts: 14
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
I know that this may not be the right place to ask this but do you think that to run this using a servlet is a crazy idea? the reason being that that way it will provide an easy way of accessing the app from any computer without insatalling any software(a part from the browser) an easy user interface to check progress at any time and change settings while the app runs etc.
If you can think of any reason why this may not be a good idea I will appreciate your comments

Thanks
 
Stan James
(instanceof Sidekick)
Ranch Hand
Posts: 8791
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
I wouldn't trigger the directory scanner from a servlet that a user hits. You want it to run all day long, and a user can't wait all day for a response so you should probably start up the directory scanner when you start the server.

You could surely have your scanner process update status somewhere in memory or on a database and then have a servlet that queries the current status. What would the servlet tell you? Maybe the number of files and bytes processed in the last hour or a histogram of the last 12 hours?
 
alberto gonzalez
Greenhorn
Posts: 14
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
I am happy with being able to provide the functionality that I need, my only concern was running the infinite loop inside a web container, just in case somebody has had problems running similar things. The user doesn't visualize the processed documents using this web app, this will only be the user interface to start it, stop it, check progress and modify settings that can be modify whilst running process. The files get moved to another system once they are processed.

Thanks
 
It is sorta covered in the JavaRanch Style Guide.
  • Post Reply Bookmark Topic Watch Topic
  • New Topic
Boost this thread!