Read Data from webpage

[Kuldeep singh ranbanka Rathore]

hi 2 all

i want to know how can i read the data from a webpage which continuously refreshes itself..
i tried but i got html source code of page that time..

[Ulf Dittmer]

What differentiates "the data" from "the html source code"? They're both the same thing, no? If you want updated HTML, you'll have to read the page again, but you may have problems if the page uses AJAX to refresh parts of itself - that's not easily amenable to screen scraping.

[Kuldeep singh ranbanka Rathore]

Sir,
for example if we talk about any website who provides you live score(which automatically gets updated) of any game, then how can read that data(say score)

[Ulf Dittmer]

My premier tool of choice for that would be a library like HtmlUnit (or jWebUnit, which is built on top of it). But again, if the page refreshes, then those libraries need to reload the page - that doesn't happen automatically. And if the live update happens through AJAX, all bets may be off.

[Henry Wong]

Kuldeep singh ranbanka Rathore wrote:Sir,
for example if we talk about any website who provides you live score(which automatically gets updated) of any game, then how can read that data(say score)

There is no one answer -- it depends on what the page is doing. And how it is getting the information from the server.

All the URL does is get the initial information for the page. This page can load tons of stuff, including javascript, flex, sl, java, xslt, etc. etc. which make calls to the server to get more information. It may also do tons of processing too, which means if you want the "score" displayed on the page, you need to know how the page application loads the information, and how it processes it for display.

Henry

[Miran Cvenkel]

red this again:

Ulf Dittmer wrote:but you may have problems if the page uses AJAX to refresh parts of itself - that's not easily amenable to screen scraping.

my opinion, pretty futile to attempt to do that, unless prowider gives you access to some web service.

[Kuldeep singh ranbanka Rathore]

So how can i know that how any website is refreshing...? any tutorial regarding this?
thanks in advance..

[Ulf Dittmer]

Miran Cvenkel wrote:my opinion, pretty futile to attempt to do that, unless prowider gives you access to some web service.

Not necessarily. HtmlUnit works fine with many common JavaScript libraries like Prototype, jQuery, Ext JS, Dojo, GWT etc. Someone not averse to tinkering might just get it to work.

[Ulf Dittmer]

Kuldeep singh ranbanka Rathore wrote:So how can i know that how any website is refreshing...?

By analyzing its source code. It might be as simple a having a Meta refresh header in which case simple reloading would work. Or it might be using JavaScript, in which case it gets trickier, and you may have to analyze the actual HTTP traffic that goes over the wire.

[Kuldeep singh ranbanka Rathore]

can you tell me about the following website?webpage
thanks for help you all........

[Ulf Dittmer]

If you want to get this up and running then you will need to become knowledgeable about HTTP and HTML; no better way to start that than to do it yourself

[Kuldeep singh ranbanka Rathore]

actually all i want to do is that i want to make an application which can fetch and show live scores from the website providing live scores.

[Ulf Dittmer]

Make sure that what you're doing is legal - most web content has a copyright that prohibits republishing.

[Kuldeep singh ranbanka Rathore]

I am a student and trying to do that only for testing purpose not commertial. By the way, thanks for advice

[William Brogden]

For understanding how a web page is constructed, I suggest the Firefox browser with plugins like Firebug that let you examine all the components.

Given the amount of money involved in sports data licensing, you may find that the designers have made it really difficult to retrieve only the scores.

Way back (1996) when Java first got started I helped make a "Instant Sports" applet that animated baseball games live while reporting scores and status of other major league games. When MLB (Major League Baseball) got an idea of how much they could get for this stuff we were priced and lawyered out.

Bill

[ujjwal soni]

Hi,

You can use com.gargoylesoftware.htmlunit.WebClient for this.

URL url=new URL("http://scores.com/espn/live/scores.html");

final HtmlPage page1 = (HtmlPage) webClient.getPage(url);

System.out.println("html code for scores >>>" + page1.asText());

//now you need to check here for a string which contains the score, read it and then disply it in your application

//first make sure weather it is legal to use data from a 3rd party website or not.

[Ulf Dittmer]

You can use com.gargoylesoftware.htmlunit.WebClient for this.

...which is what I suggested in my second post.

[Kuldeep singh ranbanka Rathore]

thanks ujjwal and ulf.....