Win a copy of Murach's Python Programming this week in the Jython/Python forum!
  • Post Reply Bookmark Topic Watch Topic
  • New Topic

crawler help  RSS feed

 
karthik mama
Ranch Hand
Posts: 34
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
dear friends can i get help of how to get the source code of a web page into a file when i present the URL dynamically i.e for example URL name=new URL("http://www.google.co.in/")
i am encountering a problem when the site needs an authentication
 
Ulf Dittmer
Rancher
Posts: 42970
73
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Assuming you're talking about BASIC authentication, then that's answered here. (Never mind that it talks about applets - it's the same for all Java clients.)
 
karthik mama
Ranch Hand
Posts: 34
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
hey thanks for replying

the code is not working

public class Crawler {

/** Creates a new instance of Crawler */
public Crawler() {
}

public static void main(String[] args) throws Exception{
URL crawlURL = new URL("http://www.google.co.in/");
URLConnection yc = crawlURL.openConnection();

sun.misc.BASE64Encoder encoder = new sun.misc.BASE64Encoder();
String encodedUserPwd =
encoder.encode("kg_b05"+":"+"*****");
yc.setRequestProperty("Proxy-Authorization", "Basic " +encodedUserPwd);

InputStream i = yc.getInputStream();
FileOutputStream fos = new FileOutputStream("file.txt");
int in=i.read();
while(in!=-1){
fos.write(in);
in=i.read();
}
fos.flush();
fos.close();
}
}


help out
 
Ulf Dittmer
Rancher
Posts: 42970
73
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Well, you didn't say anything about a proxy. A quick Google search finds a number of articles discussing how to perform programmatic web access through proxies, including authentication.
 
karthik mama
Ranch Hand
Posts: 34
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
hey thanks fr d help
 
  • Post Reply Bookmark Topic Watch Topic
  • New Topic
Boost this thread!