scanning the webpage

[Harbir Singh]

I have to write a program that will be supplied with the URL of a webpage, and it will then scan the page to get all the links to the other page and print those links. I am new to this, please if someone can give me a idea to handle this, it will be great.
Thanx
Regards
Harbir

[Scott Dunbar]

Should I reply here or over here?

I'd start with something like Apache Commons HttpClient to get the page. After that you'll likely want to use some regular expressions to look for "<a" and an href. This part isn't trivial - there are very few well written web pages out there. And you will still miss anchors that get defined in Javascript.

[Jesper de Jong]

You don't really need the Apache Jakarta HttpClient library if all you want to do is read a web page. You can use class java.net.URL for that:

URL url = new URL("http://www.somewebsite.com/index.html");
 
BufferedReader br = new BufferedReader(new InputStreamReader(url.openStream()));
 
// Read the webpage line by line
String line;
while ((line = br.readLine()) != null) {
    // Do something with the line
}
 
br.close();

If you want to parse HTML, you could use a library like this one: http://htmlparser.sourceforge.net/

[Pradhip Prakash]

This program will ask for url. You enter the correct url and press enter.

import java.net.*;
import java.io.*;

public class URLRead
{
public static void main(String[] args) throws Exception
{
BufferedReader br = new BufferedReader(new InputStreamReader(System.in));
String s;
System.out.println("Enter Your URL :");
s = br.readLine();


URL url = new URL(s);
BufferedReader in = new BufferedReader(new InputStreamReader(url.openStream()));

String inputLine;

while ((inputLine = in.readLine()) != null)
System.out.println(inputLine);

in.close();
}
}

[Stan James]

Once you have the page as a String, to dig out the links it would be good to have a solid HTML parser. I like the Quiotix Parser because it has a neat Visitor interface.

[Scott Dunbar]

Great ideas. The only reason I suggested HttpClient is for the non-trival cases - SSL, Cookies, Basic and Form based auth, etc. Harbir - alot of it depends on your requirements.