Win a copy of Murach's Python Programming this week in the Jython/Python forum!
  • Post Reply Bookmark Topic Watch Topic
  • New Topic

Web crawler or RSS Feed  RSS feed

 
Sanjoo Singh
Ranch Hand
Posts: 33
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
I need to create a program for price comparison and scheme comparison between multiple Cell service provider. Can anybody suggest any way or method to accomplish this task. Is a web crawler or RSS feed would be helpful.

After googling I found that RSS generally have news related items and in case of web crawler the program needs to be updated as the page updates. Any help regarding area would really we appreciated as I stuck here in implementing the scenario.

Thanks in advance.
 
Ulf Dittmer
Rancher
Posts: 42970
73
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Are you're saying that you don't have a reliable data source that you can use for this, and will be relying on whatever information you can find on the public web? If so, then this information probably won't be available via RSS. Scraping provider web sites using a library like HtmlUnit (or JWebUnit) seems a more feasible approach.
 
Sanjoo Singh
Ranch Hand
Posts: 33
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Thanks Dittmer,

Actually I want to create a database for storing schemes and their corresponding pricing and that can be displayed on user's request. These data should be updated automatically (through a program that I need) as per the changes done on that page.

Well, I have just visited the JWebUnit page at a glance and what I learnt is, its a testing framework. Please correct me if I am wrong and then who I can use it in my scenario.
 
Sanjoo Singh
Ranch Hand
Posts: 33
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
I have read somewhere this can be achieved using Live data feed. Any idea whats this?
 
Ulf Dittmer
Rancher
Posts: 42970
73
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
I have just visited the JWebUnit page at a glance and what I learnt is, its a testing framework. Please correct me if I am wrong and then who I can use it in my scenario.

Indeed, it's primary purpose is as a web testing extension to JUnit, but it works perfectly fine as a general-purpose library for programmatic web access.
 
  • Post Reply Bookmark Topic Watch Topic
  • New Topic
Boost this thread!