• Post Reply Bookmark Topic Watch Topic
  • New Topic
programming forums Java Mobile Certification Databases Caching Books Engineering Micro Controllers OS Languages Paradigms IDEs Build Tools Frameworks Application Servers Open Source This Site Careers Other Pie Elite all forums
this forum made possible by our volunteer staff, including ...
Marshals:
  • Campbell Ritchie
  • Jeanne Boyarsky
  • Ron McLeod
  • Paul Clapham
  • Liutauras Vilda
Sheriffs:
  • paul wheaton
  • Rob Spoor
  • Devaka Cooray
Saloon Keepers:
  • Stephan van Hulst
  • Tim Holloway
  • Carey Brown
  • Frits Walraven
  • Tim Moores
Bartenders:
  • Mikalai Zaikin

Write To A URL and Communicate With The Host Server

 
Ranch Hand
Posts: 1309
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
The java.net.URL is the package we use to read from and write to existing web pages. I have a Java program that imports the java.net.URL and reads and extracts some information from web pages. However, it is only the latter half of the task.
How does a Java program interact with; for example, cgi-bin scripts, on the server side provide that I have no idea about the contents of the cgi-bin scripts?
What I mean is that:
Many HTML pages contain forms-- text fields and other GUI objects that let us enter data to send to the server. After we type in the required information and initiate the query by clicking a button, the Web browser writes the data to the URL over the network. At the other end, a cgi-bin script (usually) on the server receives the data, processes it, and then sends us "a response", usually in the form of a new HTML page. As I stated in the beginning of this post, I have prepared a Java program to read the responding HTML page.

My question is regarding the first half of the process. How do I feed data in my Java program that initiate the query (instead of manually entering data in the text field) over the network for the host server to process? And then the Java program continues its work in reading the HTML page returned by the server? Provided that I do not know what kind of the program the hosting server uses to process the query.
 
Ranch Hand
Posts: 583
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
To make a request to a URL, URLConnection object.
Well, the same can also be used to submit data to the url. The simplest way would be to submit to the URL using query strings.. you know the "?name=..." kinda format after the url.
If there is anything more specific, please let me know.
Regds
Lupo
 
Ranch Hand
Posts: 2937
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator


How do I feed data in my Java program that initiate the query (instead of manually entering data in the text field) over the network for the host server to process?


This is more or less straightforward:
 
JiaPei Jen
Ranch Hand
Posts: 1309
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Thanks for your attention. The example can be viewed in the link below:
http://www.netsol.com/cgi-bin/whois/whois
At this web site, we can enter a domain name; say, yahoo.com, in the text field and click on the GO button. Then this HTML web page
http://www.netsol.com/cgi-bin/whois/whois?STRING=yahoo.com&SearchType=do&STRING2.x=32&STRING2.y=10
is returned by the host server. I am supposed to get the e-mail address of the Administrative Contact in that page.
I have to put the whole process in a Java program because I have to get the e-mail address of the Administrative Contact for a long list of domain names.
I have already prepared a Java program to retrieve the e-mail address of the Administrative Contact at the http://www.netsol.com/cgi-bin/whois/whois?STRING=yahoo.com&SearchType=do&STRING2.x=32&STRING2.y=10
I do not know the program that the host server uses to process the inquiry. My question is
1. how to feed domain names (of course one at a time in a loop) in a Java program for http://www.netsol.com/cgi-bin/whois/whois to receive as input? And
2. how do I know the URL of the HTML web page that is to be returned by the host server so that I can use my Java program to scan for the e-mail address of the Administrative Contact?
I am also thinking about logging into a remote web site that uses basic authentication or form based authentication and hold the session with a Java program.
Thanks in advance.
[ January 02, 2003: Message edited by: JiaPei Jen ]
[ January 02, 2003: Message edited by: JiaPei Jen ]
 
John Smith
Ranch Hand
Posts: 2937
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
I do not know the program that the host server uses to process the inquiry.

You don't need to know it whatsoever, all you need is an URL.

1. how to feed domain names (of course one at a time in a loop) in a Java program for http://www.netsol.com/cgi-bin/whois/whois to receive as input?

I already listed an example in the post above.

2. how do I know the URL of the HTML web page that is to be returned by the host server

You don't need to know that either, youjsut read the server response from the input stream using connection.getInputStream()

I am also thinking about logging into a remote web site that uses basic authentication or form based authentication and hold the session with a Java program.

This is also very straightforward, essentially you instantiate a class that incapsulates your user name and password, and a method of this class is called by your connection whenever needed:

Eugene.
[ January 03, 2003: Message edited by: Eugene Kononov ]
 
JiaPei Jen
Ranch Hand
Posts: 1309
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Hello, Eugene, thanks for your responses to my questions. I tried you code; however, something gets screwed up:
http://www.netsol.com/cgi-bin/whois/whois
is the web site where I enter the query data in the text field. Suppose I enter; say, yahoo.com, and click on the GO button, I get this HTML web page with the link shown below in the address bar:

http://www.netsol.com/cgi-bin/whois/whois?STRING=yahoo.com&SearchType=do&STRING2.x=32&STRING2.y=10
And this is the page I am going to search the information I need.
I used your code; however, the post data are not written to the URL. I only get the "host web page". And I got the HTML code of that whole page printed to the console.
I experimented a little, and found that if I enter
http://www.netsol.com/cgi-bin/whois/whois?STRING=yahoo.com in the address bar, I am directed to an error page:
http://www.netsol.com/en_US/common/error.jhtml
and if I type http://www.netsol.com/cgi-bin/whois/whois?STRING=yahoo.com&SearchType=do
in the address bar, I will be able to get the expected web page.
Therefore, I must try to append &SearchType=do to the postData in the Java program. Nonetheless, the code shown below does not seem to work:

What is the correct way to do it?
[ January 03, 2003: Message edited by: JiaPei Jen ]
[ January 04, 2003: Message edited by: JiaPei Jen ]
[ January 04, 2003: Message edited by: JiaPei Jen ]
 
John Smith
Ranch Hand
Posts: 2937
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator


postData += "?";


Your post data should start with the first parameter, not the question mark:
postData = "STRING=yahoo.com&SearchType=do"
Eugene.
[ January 05, 2003: Message edited by: Eugene Kononov ]
 
reply
    Bookmark Topic Watch Topic
  • New Topic