• Post Reply Bookmark Topic Watch Topic
  • New Topic

Deciding on an approach to connect to the web periodically to spawn a web page  RSS feed

 
Kerry Jones
Greenhorn
Posts: 5
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
We have a 4 web pages that need to be spawned every few minutes(fixed time interval). The text is then parsed and the program extracts the useful content and stores it into the database.

In the past we implemented threads that run periodically to fetch the data and sleep.

Could you please suggest any other approach that we should follow in implementing these processes or approach or do you think the current approach is good.
 
Paul Clapham
Sheriff
Posts: 22823
43
Eclipse IDE Firefox Browser MySQL Database
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
I don't understand what you mean by "spawning" a web page. In fact I don't get any picture of what your goal is at all.
 
Kerry Jones
Greenhorn
Posts: 5
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
An example: lets say parsing a finance document to parse current stock value for a particular company.



Basically the data in these web pages is updated regularly and our program needs to connect to the web page parse the data(retrieve the value of a particular field, as the program already knows the format of the web page) and update the field in database.
 
Joe Ess
Bartender
Posts: 9439
12
Linux Mac OS X Windows
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Do you have problems with the current approach? If it is working now, that's probably Good Enough
 
Kerry Jones
Greenhorn
Posts: 5
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
NO issues as such, But I just wanted to know other developers views as to which approach would have they followed for the same goal.
 
Paul Clapham
Sheriff
Posts: 22823
43
Eclipse IDE Firefox Browser MySQL Database
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
I would have just used a java.util.Timer object which ran at that specified interval and did that specified thing. Keep it simple, and If it ain't broke don't fix it, as Joe Ess suggested.

Unless this task was supposed to be embedded in some existing framework, in which case I would have to look at how to do that sort of thing in the framework.
 
Jelle Klap
Bartender
Posts: 1952
7
Eclipse IDE Java
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
So if I understand correctly the program publishes some number of different webpages at a fixed time interval and then proceeds to screen scrape those webpages to read back the information for processing?
That would certainly be a ... novel approach.
Somehow I don't think I'm getting a clear picture here
 
Kerry Jones
Greenhorn
Posts: 5
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Alright.

Thanks for the input. It is highly appreciated.
 
Paul Clapham
Sheriff
Posts: 22823
43
Eclipse IDE Firefox Browser MySQL Database
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Originally posted by Jelle Klap:
So if I understand correctly the program publishes some number of different webpages at a fixed time interval and then proceeds to screen scrape those webpages to read back the information for processing?
No, that's what I thought "spawn" meant when I originally read the question. But nothing is being spawned at all. It's just the second part of your description, the code periodically goes out to a web page and scrapes the data it downloads.
 
Kerry Jones
Greenhorn
Posts: 5
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Jelle Klap, the program does not publish the data.
It only reads the data from the web-pages.
 
Jelle Klap
Bartender
Posts: 1952
7
Eclipse IDE Java
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Originally posted by Paul Clapham:
No, that's what I thought "spawn" meant when I originally read the question. But nothing is being spawned at all. It's just the second part of your description, the code periodically goes out to a web page and scrapes the data it downloads.


Oh, so it wasn't just me
[ August 18, 2008: Message edited by: Jelle Klap ]
 
It is sorta covered in the JavaRanch Style Guide.
  • Post Reply Bookmark Topic Watch Topic
  • New Topic
Boost this thread!