How to get response content of a website

[ram gaurav]

Hi

Like through java.net.URL i am trying to connect to some site like www.google.com,now i want the response send by GOOGLE in text format not in HTML.

[Bear Bibeault]

By just hitting the google web URL, you're going to get back HTML.

If you want more control, google has a rich toolkit of web services as well as a Java API that you can use to do pretty much whatever you want.
[ March 07, 2007: Message edited by: Bear Bibeault ]

[ram gaurav]

Thanks for reply.

I have just taken example of google.com.
Like my code is :

  HttpURLConnection connection = (HttpURLConnection) new URL("http://www.yahoo.com").openConnection();
         

Now just after this it gives me error.
Can you please tell us that where are we doing wrong

  java.net.ConnectException: Connection refused: connect
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:333)
at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:195)
at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:182)
at java.net.Socket.connect(Socket.java:507)
at java.net.Socket.connect(Socket.java:457)
at sun.net.NetworkClient.doConnect(NetworkClient.java:157)
at sun.net.www.http.HttpClient.openServer(HttpClient.java:365)
at sun.net.www.http.HttpClient.openServer(HttpClient.java:477)
at sun.net.www.http.HttpClient.<init>(HttpClient.java:214)
at sun.net.www.http.HttpClient.New(HttpClient.java:287)
at sun.net.www.http.HttpClient.New(HttpClient.java:299)
at sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:792)
at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:744)
at sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:669)
at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:913)
at java.net.URLConnection.getContent(URLConnection.java:682)
at com.GetSourceServlet.doGet(GetSourceServlet.java:36)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:689)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:802)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:237)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:157)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:214)
at org.apache.catalina.core.StandardValveContext.invokeNext(StandardValveContext.java:104)
at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:520)
at org.apache.catalina.core.StandardContextValve.invokeInternal(StandardContextValve.java:198)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:152)
at org.apache.catalina.core.StandardValveContext.invokeNext(StandardValveContext.java:104)
at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:520)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:137)
at org.apache.catalina.core.StandardValveContext.invokeNext(StandardValveContext.java:104)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:118)
at org.apache.catalina.core.StandardValveContext.invokeNext(StandardValveContext.java:102)
at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:520)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
at org.apache.catalina.core.StandardValveContext.invokeNext(StandardValveContext.java:104)
at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:520)
at org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:929)
at org.apache.coyote.tomcat5.CoyoteAdapter.service(CoyoteAdapter.java:160)
at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:799)
at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.processConnection(Http11Protocol.java:705)
at org.apache.tomcat.util.net.TcpWorkerThread.runIt(PoolTcpEndpoint.java:577)
at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:683)
at java.lang.Thread.run(Thread.java:595)  

[Ulf Dittmer]

Can you ping Yahoo from where you're running this code?

Servlet engines like Tomcat often run with a security manager, which may prohibit all kinds of activity, but particularly networking (although that doesn't seem to be the problem here).

[ram gaurav]

Thanks for reply.

Actually Ping uses different protocol and HTTP is different.

Like i can simply type www.yahoo.com in my browser , it open an HTML page for , but i want the response as plain text not as HTML.

[Ulf Dittmer]

What I was asking is whether you can connect to Yahoo from the machine where you're running that code. Does that mean the answer to that is "yes"? Is it the same machine on which you're running the browser? Are there any further exception in the Tomcat logs?

[ram gaurav]

Ya i can connect to yahoo.com from my browser , but when i try to connect through my Servlet it give Connection Refused exception

[William Brogden]

I am going to guess that you have some sort of security software running on that system. Many kinds of "malware" try to connect to urls for various nefarious reasons so security software may only allow outgoing connections to be made by known programs.

Bill

[ram gaurav]

Ya , that can be the reason because we are running with some firewalls in our systems.

So how to proceed with that

[ram gaurav]

Hi

In my previous code i have add few things

  con.setRequestProperty("http.proxyHost","myProxyHost");
            con.setRequestProperty("http.proxyPort", "80");  

and now Connection refused error is not coming, but now HTTP 403 ( Forbidden : The server understood the request, but is refusing to fulfill it. Typically due to file access permissions on the server. )

Can you please tell me that why it is not allowing me and giving HTTP 403 error.

Thanks
Ram