• Post Reply Bookmark Topic Watch Topic
  • New Topic

HTTP GET request keeps getting 400 response code despite the link working perfectly fine in browser  RSS feed

 
Edwardd Lee
Ranch Hand
Posts: 33
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
I am trying to get a JSON response from Google Trends using HTTP. This is my code snippet:

This is the output:
If I type in the URL directly in my browser, Google gives me a JSON file with no problem. However, if I try to access that URL using Java, I am given a bad request. How can I solve this problem? Thanks in advance.
 
Carey Brown
Saloon Keeper
Posts: 3309
46
Eclipse IDE Firefox Browser Java MySQL Database VI Editor Windows
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Are you going through a proxy?
Have you tried some other URL that you know should work?
 
Edwardd Lee
Ranch Hand
Posts: 33
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Carey Brown wrote:Are you going through a proxy?
Have you tried some other URL that you know should work?


No I am not.
And yes. For instance, if I just parse in https://www.google.com the response code is 200.
 
Carey Brown
Saloon Keeper
Posts: 3309
46
Eclipse IDE Firefox Browser Java MySQL Database VI Editor Windows
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Have you tried it with just http:// instead of https:// ?
 
Edwardd Lee
Ranch Hand
Posts: 33
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Carey Brown wrote:Have you tried it with just http:// instead of https:// ?


Yup and it still didn't work...
 
Paul Clapham
Sheriff
Posts: 22819
43
Eclipse IDE Firefox Browser MySQL Database
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
When I looked at Google's site for the Trends API, I found a link to the API's Terms of Service. It said that you shouldn't try to access the Trends data via anything other than their API. And you're trying to access the data via Java.

So it appears that there's some code at the back end which looks for requests like yours which come from unauthorized code. Presumably there's a way to make your requests look like they come from the authorized API but I don't know what that might be.
 
Edwardd Lee
Ranch Hand
Posts: 33
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Paul Clapham wrote:When I looked at Google's site for the Trends API, I found a link to the API's Terms of Service. It said that you shouldn't try to access the Trends data via anything other than their API. And you're trying to access the data via Java.

So it appears that there's some code at the back end which looks for requests like yours which come from unauthorized code. Presumably there's a way to make your requests look like they come from the authorized API but I don't know what that might be.


I am actually trying to model my code after a similar Python code which works. It seems that the Python code can access Google Trend's API without issues, but when I tried to write it in Java there are issues...
 
Edwardd Lee
Ranch Hand
Posts: 33
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
I just tried to use URLEncoder to encode my query. The results was very interesting - now the response code is 301 instead of 400. However, the results still turn out to be the wrong file, no matter if I set con.setInstanceFollowRedirects(); to true or false.

The output is now:
If I type the encoded URL in the browser myself, I still get the correct JSON file, not the <HTML><HEAD><TITLE>Moved Permanently</TITLE></HEAD>... file in the output. This is a very weird behavior, and I'm not sure if this is because of Google or my code...
 
Ron McLeod
Bartender
Posts: 1602
232
Android Angular Framework Eclipse IDE Java Linux MySQL Database Redhat TypeScript
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
It looks like a query string encoding issue.  I quickly encoded the space and quote characters and got a 200 response back.  I didn't bother trying to retrieve the returned body.
 
Ron McLeod
Bartender
Posts: 1602
232
Android Angular Framework Eclipse IDE Java Linux MySQL Database Redhat TypeScript
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Edwardd Lee wrote:I just tried to use URLEncoder to encode my query. The results was very interesting - now the response code is 301 instead of 400. However, the results still turn out to be the wrong file, no matter if I set con.setInstanceFollowRedirects(); to true or false.

HttpURLConnection may not follow redirects from http to https.

Change your URL to from http://trends.google.com/trends/... to https://trends.google.com/trends/...

Also, remove setInstanceFollowRedirects().  The default behaviour is to follow.
 
Edwardd Lee
Ranch Hand
Posts: 33
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Ron McLeod wrote:It looks like a query string encoding issue.  I quickly encoded the space and quote characters and got a 200 response back.  I didn't bother trying to retrieve the returned body.


Oh that does work! Thank you! Looks like I need to learn knowledge about encoding now.
 
It is sorta covered in the JavaRanch Style Guide.
  • Post Reply Bookmark Topic Watch Topic
  • New Topic
Boost this thread!