web crawler compilation problem

[bryan lim]

hi all,

i was looking at the web crawler in hereand source code here. i tried running it but got following error:

Exception in thread "main" java.lang.Error: Unresolved compilation problem:

at crawler.WebCrawler.main(WebCrawler.java:32)

i corrected the old main to "public static void main(String[] args) {" but still got error.

please advice. i am using eclipse.

[Henry Wong]

Exception in thread "main" java.lang.Error: Unresolved compilation problem:

at crawler.WebCrawler.main(WebCrawler.java:32)

Eclipse has the capability to run something that didn't compile right. This just means that you have a compile error, tried to run it anyway, and ran right into the code that didn't compile. Basically, the solution is to fix the compile errors before you run it.

i corrected the old main to "public static void main(String[] args) {" but still got error.

What was wrong with the old main() ? It looks fine to me. And all you seem to have done was change the name of the argument -- what is that suppose to do?

please advice. i am using eclipse.

I would suggest that you fix the code that is labelled in red. Eclipse marks all error in red, and on the left of the editor, provides a ballon, with a tooltip, telling what is wrong with the line.

Henry

[bryan lim]

thank you for your reply. but there is no red line under the line giving error and

i only got this message as an error.

Exception in thread "main" java.lang.Error: Unresolved compilation problem:

at crawler.WebCrawler.main(WebCrawler.java:32)

[Henry Wong]

thank you for your reply. but there is no red line under the line giving error and

If I wasn't clear before -- ignore the error that is printed when the program is ran. It is useless here. Scroll down the whole file. Find the lines marked in red. And fixed them all.

Do *not* run the program again until all the red marks are gone!! Running a program with compile errors don't give you any errors that are usable -- fix all the compile errors.

Henry

[bryan lim]

thank you for your prompt reply again.

is there anything wrong with the importing statement(referring to the print screen)? because eclipses never underline any line and i don think there is any problem with the code as well.

i got the code from java tutorial here

[Henry Wong]

Well, when you put the pointer on the left ballon, what does the tooltip say? And how do you suppose to fix that?


BTW, after you fix this, you should get five red errors appearing... which can be all fixed together by simply fixing the first one.... so don't panic.

Henry

[bryan lim]

Ok! thank you henry for your time. really appreciate it and the code is working now.

just added package crawler in .

[bryan lim]

but i have another question.

the program don't seems to be working. i typed a few url and it never show any result.

anybody knows why?

i have included the code here. thank you

package crawler;

import java.applet.Applet;
import java.text.*;
import java.awt.*;
import java.awt.event.*;
import java.util.*;
import java.net.*;
import java.io.*;
import java.awt.List;

public class WebCrawler extends Applet implements ActionListener, Runnable {
    public static final String SEARCH = "Search";
    public static final String STOP = "Stop";
    public static final String DISALLOW = "Disallow:";
    public static final int    SEARCH_LIMIT = 1000;

    Panel   panelMain;
    List    listMatches;
    Label   labelStatus;

    // URLs to be searched
    Vector vectorToSearch;
    // URLs already searched
    Vector vectorSearched;
    // URLs which match
    Vector vectorMatches;

    Thread searchThread;

    TextField textURL;
    Choice    choiceType;
    
    public static void main(String[] args) {

        Frame f = new Frame("WebFrame");
        WebCrawler applet = new WebCrawler();
        f.add("Center", applet);

/*  Behind a firewall set your proxy and port here!
*/
        Properties props= new Properties(System.getProperties());
        props.put("http.proxySet", "true");
        props.put("http.proxyHost", "webcache-cup");
        props.put("http.proxyPort", "8080");

        Properties newprops = new Properties(props);
        System.setProperties(newprops);
/**/


        applet.init();
        applet.start();
        f.pack();
        f.show();
}

    public void init() {

 // set up the main UI panel
 panelMain = new Panel();
 panelMain.setLayout(new BorderLayout(5, 5));

 // text entry components
 Panel panelEntry = new Panel();
 panelEntry.setLayout(new BorderLayout(5, 5));

 Panel panelURL = new Panel();
 panelURL.setLayout(new FlowLayout(FlowLayout.LEFT, 5, 5));
 Label labelURL = new Label("Starting URL: ", Label.RIGHT);
 panelURL.add(labelURL);
 textURL = new TextField("", 40);
 panelURL.add(textURL);
 panelEntry.add("North", panelURL);

 Panel panelType = new Panel();
 panelType.setLayout(new FlowLayout(FlowLayout.LEFT, 5, 5));
 Label labelType = new Label("Content type: ", Label.RIGHT);
 panelType.add(labelType);
 choiceType = new Choice();
 choiceType.addItem("text/html");
 choiceType.addItem("audio/basic");
 choiceType.addItem("audio/au");
 choiceType.addItem("audio/aiff");
 choiceType.addItem("audio/wav");
 choiceType.addItem("video/mpeg");
 choiceType.addItem("video/x-avi");
 panelType.add(choiceType);
 panelEntry.add("South", panelType);

 panelMain.add("North", panelEntry);

 // list of result URLs
 Panel panelListButtons = new Panel();
 panelListButtons.setLayout(new BorderLayout(5, 5));

 Panel panelList = new Panel();
 panelList.setLayout(new BorderLayout(5, 5));
 Label labelResults = new Label("Search results");
 panelList.add("North", labelResults);
 Panel panelListCurrent = new Panel();
 panelListCurrent.setLayout(new BorderLayout(5, 5));
 listMatches = new List(10);
 panelListCurrent.add("North", listMatches);
 labelStatus = new Label("");
 panelListCurrent.add("South", labelStatus);
 panelList.add("South", panelListCurrent);

 panelListButtons.add("North", panelList);

 // control buttons
 Panel panelButtons = new Panel();
 Button buttonSearch = new Button(SEARCH);
 buttonSearch.addActionListener(this);
 panelButtons.add(buttonSearch);
 Button buttonStop = new Button(STOP);
 buttonStop.addActionListener(this);
 panelButtons.add(buttonStop);

 panelListButtons.add("South", panelButtons);

 panelMain.add("South", panelListButtons);

 add(panelMain);
 setVisible(true);

 repaint(); 

 // initialize search data structures
 vectorToSearch = new Vector();
 vectorSearched = new Vector();
 vectorMatches = new Vector();

 // set default for URL access
 URLConnection.setDefaultAllowUserInteraction(false);
    }

    public void start() {
    }

    public void stop() {
 if (searchThread != null) {
     setStatus("stopping...");
     searchThread = null;
 }
    }

    public void destroy() {
    }

    boolean robotSafe(URL url) {
 String strHost = url.getHost();

 // form URL of the robots.txt file
 String strRobot = "http://" + strHost + "/robots.txt";
 URL urlRobot;
 try { 
     urlRobot = new URL(strRobot);
 } catch (MalformedURLException e) {
     // something weird is happening, so don't trust it
     return false;
 }

 String strCommands;
 try {
     InputStream urlRobotStream = urlRobot.openStream();

     // read in entire file
     byte b[] = new byte[1000];
     int numRead = urlRobotStream.read(b);
     strCommands = new String(b, 0, numRead);
     while (numRead != -1) {
  if (Thread.currentThread() != searchThread)
      break;
  numRead = urlRobotStream.read(b);
  if (numRead != -1) {
      String newCommands = new String(b, 0, numRead);
      strCommands += newCommands;
  }
     }
     urlRobotStream.close();
 } catch (IOException e) {
     // if there is no robots.txt file, it is OK to search
     return true;
 }

 // assume that this robots.txt refers to us and 
 // search for "Disallow:" commands.
 String strURL = url.getFile();
 int index = 0;
 while ((index = strCommands.indexOf(DISALLOW, index)) != -1) {
     index += DISALLOW.length();
     String strPath = strCommands.substring(index);
     StringTokenizer st = new StringTokenizer(strPath);

     if (!st.hasMoreTokens())
  break;
     
     String strBadPath = st.nextToken();

     // if the URL starts with a disallowed path, it is not safe
     if (strURL.indexOf(strBadPath) == 0)
  return false;
 }

 return true;
    }

    public void paint(Graphics g) {
       //Draw a Rectangle around the applet's display area.
       g.drawRect(0, 0, getSize().width - 1, getSize().height - 1);

 panelMain.paint(g);
 panelMain.paintComponents(g);
 // update(g);
 // panelMain.update(g);
    }

    public void run() {
 String strURL = textURL.getText();
 String strTargetType = choiceType.getSelectedItem();
 int numberSearched = 0;
 int numberFound = 0;

 if (strURL.length() == 0) {
     setStatus("ERROR: must enter a starting URL");
     return;
 }

 // initialize search data structures
 vectorToSearch.removeAllElements();
 vectorSearched.removeAllElements();
 vectorMatches.removeAllElements();
 listMatches.removeAll();

 vectorToSearch.addElement(strURL);

 while ((vectorToSearch.size() > 0) 
   && (Thread.currentThread() == searchThread)) {
     // get the first element from the to be searched list
     strURL = (String) vectorToSearch.elementAt(0);

     setStatus("searching " + strURL);

     URL url;
     try { 
  url = new URL(strURL);
     } catch (MalformedURLException e) {
  setStatus("ERROR: invalid URL " + strURL);
  break;
     }

     // mark the URL as searched (we want this one way or the other)
     vectorToSearch.removeElementAt(0);
     vectorSearched.addElement(strURL);

     // can only search http: protocol URLs
     if (url.getProtocol().compareTo("http") != 0) 
  break;

     // test to make sure it is before searching
     if (!robotSafe(url))
  break;

     try {
  // try opening the URL
  URLConnection urlConnection = url.openConnection();

  urlConnection.setAllowUserInteraction(false);

  InputStream urlStream = url.openStream();
  String type 
    = urlConnection.guessContentTypeFromStream(urlStream);
  if (type == null)
      break;
  if (type.compareTo("text/html") != 0) 
      break;

  // search the input stream for links
  // first, read in the entire URL
  byte b[] = new byte[1000];
  int numRead = urlStream.read(b);
  String content = new String(b, 0, numRead);
  while (numRead != -1) {
      if (Thread.currentThread() != searchThread)
   break;
      numRead = urlStream.read(b);
      if (numRead != -1) {
   String newContent = new String(b, 0, numRead);
   content += newContent;
      }
  }
  urlStream.close();

  if (Thread.currentThread() != searchThread)
      break;

  String lowerCaseContent = content.toLowerCase();

  int index = 0;
  while ((index = lowerCaseContent.indexOf("<a", index)) != -1)
  {
      if ((index = lowerCaseContent.indexOf("href", index)) == -1) 
   break;
      if ((index = lowerCaseContent.indexOf("=", index)) == -1) 
   break;
      
      if (Thread.currentThread() != searchThread)
   break;

      index++;
      String remaining = content.substring(index);

      StringTokenizer st 
        = new StringTokenizer(remaining, "\t\n\r\">#");
      String strLink = st.nextToken();

      URL urlLink;
      try {
   urlLink = new URL(url, strLink);
   strLink = urlLink.toString();
      } catch (MalformedURLException e) {
   setStatus("ERROR: bad URL " + strLink);
   continue;
      }

      // only look at http links
      if (urlLink.getProtocol().compareTo("http") != 0)
   break;

      if (Thread.currentThread() != searchThread)
   break;

      try {
   // try opening the URL
   URLConnection urlLinkConnection 
     = urlLink.openConnection();
   urlLinkConnection.setAllowUserInteraction(false);
   InputStream linkStream = urlLink.openStream();
   String strType 
     = urlLinkConnection.guessContentTypeFromStream(linkStream);
   linkStream.close();

   // if another page, add to the end of search list
   if (strType == null)
       break;
   if (strType.compareTo("text/html") == 0) {
       // check to see if this URL has already been 
       // searched or is going to be searched
       if ((!vectorSearched.contains(strLink)) 
         && (!vectorToSearch.contains(strLink))) {

    // test to make sure it is robot-safe!
    if (robotSafe(urlLink))
        vectorToSearch.addElement(strLink);
       }
   }

   // if the proper type, add it to the results list
   // unless we have already seen it
   if (strType.compareTo(strTargetType) == 0) {
       if (vectorMatches.contains(strLink) == false) {
    listMatches.add(strLink);
    vectorMatches.addElement(strLink);
    numberFound++;
    if (numberFound >= SEARCH_LIMIT)
        break;
       }
   }
      } catch (IOException e) {
   setStatus("ERROR: couldn't open URL " + strLink);
   continue;
      }
  }
     } catch (IOException e) {
  setStatus("ERROR: couldn't open URL " + strURL);
  break;
     }

     numberSearched++;
     if (numberSearched >= SEARCH_LIMIT)
  break;
 }

 if (numberSearched >= SEARCH_LIMIT || numberFound >= SEARCH_LIMIT)
     setStatus("reached search limit of " + SEARCH_LIMIT);
 else
     setStatus("done");
 searchThread = null;
 // searchThread.stop();
    }

    void setStatus(String status) {
 labelStatus.setText(status);
    }

    public void actionPerformed(ActionEvent event) {
 String command = event.getActionCommand();

 if (command.compareTo(SEARCH) == 0) {
     setStatus("searching...");

     // launch a thread to do the search
     if (searchThread == null) {
  searchThread = new Thread(this);
     }
     searchThread.start();
 }
 else if (command.compareTo(STOP) == 0) {
     stop();
 }
    }


}
    

[Campbell Ritchie]

That is really old code, before Java 1.2. You have to change both references to List to java.awt.List; java.util.List hadn't been introduced yet.
I suggest you find a newer web crawler with a decent explanation of how it works.

[Henry Wong]

This code looks to be a testing webcrawler, for a particular location, and for particular sites. It doesn't look like it was meant to be a general purpose crawler.

First of all, this code...

Properties props= new Properties(System.getProperties());
props.put("http.proxySet", "true");
props.put("http.proxyHost", "webcache-cup");
props.put("http.proxyPort", "8080");

Properties newprops = new Properties(props);
System.setProperties(newprops);

Sets an http proxy -- which implies that the code is supposed to run from a location that is behind a particular firewall.

Second of all, this code...

boolean robotSafe(URL url) {
    String strHost = url.getHost();

    // form URL of the robots.txt file
    String strRobot = "http://" + strHost + "/robots.txt";
    etc....

Checks and confirms a robot.txt file to double check if it is allowed to crawl the site and how to crawl the site (I guess this was a growing standard years ago). Regardless, many sites, like yahoo, doesn't have this file, so the crawling will abort. You should check to see if this may be the cause in your case.

My guess is, this code was probably used for training, and to be run only from the training site, and only to target specific testing sites.

Henry

[bryan lim]

thank you henry and Campbell Ritchie !

does anybody has a site that teach robots or web crawler to recommend? thank you again.

[Ulf Dittmer]

Some of these may fit the bill: http://java-source.net/open-source/crawlers

[bryan lim]

thank you!! i learn so much from coderanch.!!

i am wondering can i ask gwt question in coderanch?? why don't the forum open a section on gwt?? it will be great !!

[Martijn Verburg]

We basically open new forums once we get several questions on a new topic. A this stage we haven't seen enough Q's on GWT for it to warrant it's own forum. So in short please post GWT questions to the "Application Frameworks" forum