Problem grabbing RSS from a website and displaying it from servlet

[William Farrugia]

Hi,

I'm having some problems when trying to grab RSS feed from a website and display it from a servlet. My servlet runs perfectly on my development machine at home, but fails with this error on the server (CentOS)

java.io.IOException: Server returned HTTP response code 400 for URL http://www.timesofmalta.com/rss
 at gnu.java.net.protocol.http.HTTPURLConnection.getInputStream(libgcj.so.7rh)
 at subpack.GetRss.getFeeds(GetRss.java:21) 
at getnews.processRequest(getnews.java:24) 
at getnews.doGet(getnews.java:40) 
at javax.servlet.http.HttpServlet.service(tomcat5-servlet-2.4-api-5.5.23.jar.so) 
at javax.servlet.http.HttpServlet.service(tomcat5-servlet-2.4-api-5.5.23.jar.so) 
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(catalina-5.5.23.jar.so) 
at org.apache.catalina.core.ApplicationFilterChain.doFilter(catalina-5.5.23.jar.so) 
at org.apache.catalina.core.StandardWrapperValve.invoke(catalina-5.5.23.jar.so) 
at org.apache.catalina.core.StandardContextValve.invoke(catalina-5.5.23.jar.so) 
at org.apache.catalina.core.StandardHostValve.invoke(catalina-5.5.23.jar.so) 
at org.apache.catalina.valves.ErrorReportValve.invoke(catalina-5.5.23.jar.so)
 at org.apache.catalina.core.StandardEngineValve.invoke(catalina-5.5.23.jar.so)
 at org.apache.catalina.connector.CoyoteAdapter.service(catalina-5.5.23.jar.so) 
at org.apache.coyote.http11.Http11Processor.process(tomcat-http-5.5.23.jar.so) 
at org.apache.coyote.http11.Http11BaseProtocol$Http11ConnectionHandler.processConnection(tomcat-http-5.5.23.jar.so) 
at org.apache.tomcat.util.net.PoolTcpEndpoint.processSocket(tomcat-util-5.5.23.jar.so) 
at org.apache.tomcat.util.net.LeaderFollowerWorkerThread.runIt(tomcat-util-5.5.23.jar.so)
 at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(tomcat-util-5.5.23.jar.so) 
at java.lang.Thread.run(libgcj.so.7rh)

My code is as follows:

    public NewsItem[] getFeeds() throws MalformedURLException, IOException, SAXException{
        URLConnection u = new URL(url).openConnection();
        BufferedReader br = new BufferedReader(new InputStreamReader(u.getInputStream()));
        String inputLine, feed = "";
        while ((inputLine = br.readLine()) != null) feed += inputLine;
        br.close();

        Vector<NewsItem> v = new Vector();

        DOMParser parser = new DOMParser();
        parser.parse(new InputSource(new StringReader(feed)));
        Document doc = parser.getDocument();
        Node root  = doc.getChildNodes().item(0);
        Node channel = root.getChildNodes().item(1);
        NodeList list = channel.getChildNodes();

        for (int i = 0; i < list.getLength(); i++){
            Node item = list.item(i);
            if (item.getNodeName().equalsIgnoreCase("item")){
                NodeList contents = item.getChildNodes();
                v.add(new NewsItem(contents));
            }            
        }
        NewsItem[] all = new NewsItem[v.size()];
        for (int i = 0; i < v.size(); i++){
            all[i] = v.elementAt(i);
        }
        return all;
    }

The website I'm trying to access is http://www.timesofmalta.com/rss

At home I'm running Tomcat 6.0 and on the server Tomcat 5.

At first I suspected the problem might be policy related. (Happened to me on a different server in the past) so I tried to grant access

grant { permission java.net.SocketPermission "timesofmalta.com:80", "connect, accept, listen"; };

But that just didn't work.

Any help appreciated

William

[Ulf Dittmer]

I don't think it's a permission problem - the request clearly goes out, but is considered incorrect by the server (that's what a 400 response means). A "curl http://www.timesofmalta.com/rss" retrieves the feed fine for me, so this is odd.

By the way, the feed is in UTF-8, so unless your platform default encoding is UTF-8 (and it may well not be), new InputStreamReader(u.getInputStream()) will get you into trouble; use new InputStreamReader(u.getInputStream(), "UTF-8") instead.

You could also use the ROME library; it's the standard Java library for dealing with RSS.

[William Farrugia]

Hello!

Thanks for your replied. I tried adding UTF-8 but didn't solve the problem. It still works on my platform but it doesn't on the server. However you did put me in the right direction. Convinced the policies were OK, I tried switching off IPTables on the server, and guess what... it worked!

So now I will try and fix the iptables instead.

Thanks a lot for your help

William

[Ulf Dittmer]

You're welcome.

I tried adding UTF-8 but didn't solve the problem.

It wasn't supposed to; that was just an additional problem I noticed with the code that needed to be addressed.

[William Farrugia]

Ulf Dittmer wrote:It wasn't supposed to; that was just an additional problem I noticed with the code that needed to be addressed.

Sorry I misunderstood then

I had some 8080 to port 80 redirections that were mistakenly added to the output chain on IP tables. I commented out

# A OUTPUT -p tcp -m tcp --dport 80 -j REDIRECT --to-ports 8080 # A OUTPUT -d 127.0.0.1 -p tcp -m tcp --dport 80 -j REDIRECT --to-ports 8080

and it works.

Thanks and regards

William