URLConnection problem

[Chris Mooring]

Hi,
I am having a problem connecting to a URL from within a servlet.
The URL I am trying to connect to is a CGI Program that has some parameters passed as part of the URL (as in a GET) and a few FORM parameters (as in a POST). If I access the URL directly through a browser, everything works perfectly every time. However, when I try to access the URL from a servlet, it doesn't work the first time. Then when I hit the browsers refresh button, it runs the same code, but this time it works !
The CGI program is called using a POST method.
Here is what I am doing in my servlet ;
- Create a URL object with the parameter I got from the form
- Create a URLConnection object by opening a connection on my URL object
- setDoOutput(true) and setDoInput(true) on my URLConnection object
- Create a DataOutputStream (dos) to my URLconnection
- While my requestParams.hasMoreElements(), dos.writeBytes
(URLEncoder.encode(Param)) + "=" + URLEncoder.encode(req.getParameter(Param))
- I then flush and close the DataOutPutStream (dos)
- I then connect to the input stream of the URLConnection object and recieve the HTML output.
- I set res.setContentType("text/html");
- I print the HTML output to the output stream of my servlets
HTTPServletResponse
Does any of this seem incorrect or incomplete ? Why would it work fine in a browser and not in a servlet ? Should I have some sort of delay before connecting to the URL and passing through the form parameters ? Does anyone know exacly how a browser would handle reuests ?
I have been working on this problem for a while now and I am completely stumped.
I would appreciate any help I can get on this one.
Thanks,
Chris

[Sean MacLean]

Try setting the content-type earlier in the process. Sometimes this can be the cause of weird problems when you messing with I/O to a client from a servlet. Otherwise, I'd have to run the code to delve any deeper. Good luck.
Sean

[Marya Doery]

I've never done this; I'd be interested to know why you do this.
If you are just trying to have the output of a cgi go to your browser, you could use this solution that I found in the BEA user group:
===========
java.net.URL url = new java.net.URL(urlname); // url name is full String name of website, "http://whatever.com/something.cgi"
BufferedReader in = new BufferedReader(new InputStreamReader(url.openStream()));
StringBuffer sb = new StringBuffer();
int numChar = 256;
char[] cbuf = new char[numChar];
int count = 0;
try {
while ((count = in.read(cbuf, 0, numChar)) > -1) {
sb.append(cbuf, 0, count);
}
}finally{
in.close();
}
// then write sb to the response's PrintWriter
Anyway the one thing I notice is that in your last step you don't mention that you close the PrintWriter. I don't know how important that is if you didn't do it. You said "I print the HTML output to the output stream of my servlet's HttpServletResponse". I assume you do something like this:
response.setContentType("text/html");
PrintWriter pw = response.getWriter();
// code to write out
Did you close PrintWriter? - pw.close();

[Chris Mooring]

Originally posted by Marya Doery:
[B]I've never done this; I'd be interested to know why you do this.
<snip>
Hi Marya,
The reason I am writing this servlet is to act as a redirection for incoming requests. The request is made to the servlet with enough information for me to figure out where (which machine, port etc), to re-direct the incoming request to. The reason I am re-directing via a servlet, is because for certain requests, I need to do some suthentication to make sure they are allowed to view the data they are requesting. Therefore my servlet examines an incoming request, if it is of a certain type, then it will be re-directed to an authorisation servlet before coming being directed to the intended destination.
With regards to my problem, if a URL is accessed directly (ie without going through my Redirection Servlet), then everything will work fine. Things still don't go quite right when we try to access the same URL through my servlet. Therefore we are getting in a contractor to look at the problem. I have already spent 3 weeks looking at it and have run out of things to try. I'm not even sure if the problem lies in my servlet or in the CGI program.
Thanks for your reply,
Chris

[Sean MacLean]

Have a look at these sites. Looking at what you are trying to accomplish, I'm tempted to suggest a new approach all together.
http://interconinternational.com/examples/simple_servlets/redirect.html
or here http://www.esus.com/javaindex/j2ee/servlets/servletredirect.html
These are much simpler models that should do what you're asking.
Sean

[Chris Mooring]

Hi Sean,
Thanks for your reply. I had a look at those sites you suggested but unfortunately I don't think it is that simple. The machines I want to re-direct to may possibly be behind our firewall and may not be able to serve pages directly to a client out in the internet.
Also, this servlet needs to be able to handle form data and I don't think the examples allow form data (if in a POST method).
I will keep you posted on how my problem turns out.
Regards,
Chris

[Carl Trusiak]

Hi Chris,
If all the parameters you need are already included in the request, you can use a similar stratagy as forward. Include!
RequestDispatcher.include(HttpServletRequest, HttpServletResponse) This will include the output from whereever in your response bsack to the client.
------------------
Hope This Helps
Carl Trusiak, SCJP2

[Mohamed Yousuff]

Hi Ms.Chris Mooring,
I had a very similar problem recently but it is quite simple to solve. See the code below. I assume that you are trying to access the url( cgi program ) which will be called using the post method.

response.setContentType("text/html") ;
String site = "http://www.java.com/program.cgi" ;
String qstring = "" ;

URL u = new URL( site ) ;
URLConnection uc = u.openConnection() ;
uc.setDoOutput( true ) ;
PrintWriter pw = new PrintWriter( uc.getOutputStream() ) ;
Enumeration e = request.getParameterNames() ;
while ( e.hasMoreElements() ){
String pName = request.getParameter((String)e.nextElement()) ;
String pvalue = request.getParameter( pName ) ;
qstring += pName + "=" + URLEncoder.encode( pvalue ) + "&" ;
}
pw.println( qstring ) ;
pw.flush() ;
pw.close() ;
String result = "" ;
BufferedReader br = new BufferedReader( new InputStreamReader(uc.getInputStream() ) ;
while( (result=br.readLine()) != null )
response.getWriter().println( result ) ;
response.flush() ;
response.close() ;

The above code could be used for sending post request to any site. The code is written in a way to be used for any requirements. This will solve your problem.
There are some rules for sending the post request by URL or URLConnection object which you might not have followed. That may be the reason why your code is not working.

Originally posted by Chris Mooring:
Hi,
I am having a problem connecting to a URL from within a servlet.
The URL I am trying to connect to is a CGI Program that has some parameters passed as part of the URL (as in a GET) and a few FORM parameters (as in a POST). If I access the URL directly through a browser, everything works perfectly every time. However, when I try to access the URL from a servlet, it doesn't work the first time. Then when I hit the browsers refresh button, it runs the same code, but this time it works !
The CGI program is called using a POST method.
Here is what I am doing in my servlet ;
- Create a URL object with the parameter I got from the form
- Create a URLConnection object by opening a connection on my URL object
- setDoOutput(true) and setDoInput(true) on my URLConnection object
- Create a DataOutputStream (dos) to my URLconnection
- While my requestParams.hasMoreElements(), dos.writeBytes
(URLEncoder.encode(Param)) + "=" + URLEncoder.encode(req.getParameter(Param))
- I then flush and close the DataOutPutStream (dos)
- I then connect to the input stream of the URLConnection object and recieve the HTML output.
- I set res.setContentType("text/html");
- I print the HTML output to the output stream of my servlets
HTTPServletResponse
Does any of this seem incorrect or incomplete ? Why would it work fine in a browser and not in a servlet ? Should I have some sort of delay before connecting to the URL and passing through the form parameters ? Does anyone know exacly how a browser would handle reuests ?
I have been working on this problem for a while now and I am completely stumped.
I would appreciate any help I can get on this one.
Thanks,
Chris

------------------

[Chris Mooring]

Hi,
Your post is actually exactly the same as what I am doing (more or less). I changed my code so that it would look exactly the same as yours but unfortunately I still get the same result.
We hada contractor come and look at the problem today and it seems to be something happening in the CGI program I am trying to connect to. It somehow seems to lose its place (backend variables - not form data), but re-gains it after a refresh. I don't understand how this is possible, but we are working on it.
I also don't understand why everything works fine when a browser accesses the CGI program directly, but when my servlet does, it doesn't work until the page is re-freshed and the data re-posted. Hopefully it will all become clear soon and I'll let you know what the problem is (just in case it happens to someone else).
Thanks again for everyones replies,
Chris