Win a copy of TensorFlow 2.0 in Action this week in the Artificial Intelligence and Machine Learning forum!
  • Post Reply Bookmark Topic Watch Topic
  • New Topic
programming forums Java Mobile Certification Databases Caching Books Engineering Micro Controllers OS Languages Paradigms IDEs Build Tools Frameworks Application Servers Open Source This Site Careers Other all forums
this forum made possible by our volunteer staff, including ...
Marshals:
  • Campbell Ritchie
  • Liutauras Vilda
  • Paul Clapham
  • Bear Bibeault
  • Jeanne Boyarsky
Sheriffs:
  • Ron McLeod
  • Tim Cooke
  • Devaka Cooray
Saloon Keepers:
  • Tim Moores
  • Tim Holloway
  • Jj Roberts
  • Stephan van Hulst
  • Carey Brown
Bartenders:
  • salvin francis
  • Scott Selikoff
  • fred rosenberger

URLConnection get headers

 
Greenhorn
Posts: 22
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
I am trying to connect to National Stock Exchange India to get some Historical data. Previously it was easy task. Now they have added cookies and certain header fields, I am unable to get headers with following code. I tried for Google and I got header fields. What I am doing wrong




output i am getting is length = 0  
for www.google.com I am getting headers
 
Sheriff
Posts: 3207
476
Android Eclipse IDE TypeScript Redhat MicroProfile Quarkus Java Linux
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Some sites may refuse to service your request if certain expected HTTP headers are missing.  User-Agent is probably the most common one - the site may use your browser/client type to tailor the response that it returns to you.

Here's some suggestions:
   - Add a User-Agent header to your request
   - Add a Accept header to your request

Also, use HttpsURLConnection rather than the more general URLConnection.

For example:
 
S Raman
Greenhorn
Posts: 22
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Thanks Ron McLeod

I did get the headers

I am not able to download  and webscrap the quotes from NSEIndia  the Problem I noticed is when i visit the site I get a cookie name "bm_sv" and If I do set cookies bm_sv I ma able to download

However when the above code is used  I do get headers with cookies but the cookie by the name bm_sv is missing
how the server achieves .  I download page only 2 times per day and I am not requesting contionusly the site.

Can I get programatically bm_sv cookie
 
Ron McLeod
Sheriff
Posts: 3207
476
Android Eclipse IDE TypeScript Redhat MicroProfile Quarkus Java Linux
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator

S Raman wrote:I am not able to download  and webscrap the quotes from NSEIndia


I don't know anything specifically about NDEIndia, but I would assume that they have a webservice API to programmatically access site data.  Have you looked at that?
 
S Raman
Greenhorn
Posts: 22
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Thanks for suggestions  l am contacting the website
 
I have gone to look for myself. If I should return before I get back, keep me here with this tiny ad:
Building a Better World in your Backyard by Paul Wheaton and Shawn Klassen-Koop
https://coderanch.com/wiki/718759/books/Building-World-Backyard-Paul-Wheaton
reply
    Bookmark Topic Watch Topic
  • New Topic