Win a copy of Murach's Python Programming this week in the Jython/Python forum!
  • Post Reply Bookmark Topic Watch Topic
  • New Topic

java.net.HttpURLConnection retrieving a fair amount of data  RSS feed

 
Dave Robbins
Ranch Hand
Posts: 131
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Hello All

I have a servlet that needs to generate graphs of weather data
I'm using a HttpURLConnection to fetch what I need to generate the graph
I want to get the high temps for a particular zipcode for a whole year which means I get a file about 30K in size (it has other weather data included)
I do a little loop



to wait till all the data come back and then read it and start processing it
but I consistently get about 8080 bytes of data, a quarter of what I expect
I think the code is being broken into packets and I'm just getting the first one, does that make sense?
if so how can I get around this?
the guts of the code follow



TIA
Dave
 
Steve Luke
Bartender
Posts: 4181
22
IntelliJ IDE Java Python
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
The output from a web site is almost always buffered in some manner. You have a few false assumptions in your code. The first:

The response code being 200 does not signal completion of the page. It simply means the request was successful. The response code is usually pretty early in a response (it is a header).

Second, you assume that a single read from the input is enough to get all the data. Because the response will usually be buffered it will show up in chunks. So you have to be prepared to read it in chunks:

Or something to that affect. This will keep on attempting to read chunks from the stream until there is nothing left to read. It appends the number of characters actually read to an StringBuilder as it goes along.
 
Dave Robbins
Ranch Hand
Posts: 131
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
thanks
I thought the answer was something to that affect
I'll let you know how it works out

Dave
 
Dave Robbins
Ranch Hand
Posts: 131
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
beautiful
one tweak

change this line



to this



in case somebody else comes along and uses your tip
works great!!
Thanks

Dave
 
Steve Luke
Bartender
Posts: 4181
22
IntelliJ IDE Java Python
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Dave Robbins wrote:beautiful
one tweak

change this line



to this



in case somebody else comes along and uses your tip
works great!!
Thanks

Dave


Are you sure? I think I might have screwed up more than that. The three parameters are:
cbuf - Destination buffer
off - Offset at which to start storing characters
len - Maximum number of characters to read

The offset has to do with the offset in the buffer and the length is the total number of bytes to read. So my original mistake was to think the offset was the offset in the input stream I was reading. I would change the reading line to:

So that I am constantly overwriting the start of the buffer up to a maximum of the buffer length. This lets the copy over to the output work correctly:

Since I am always copying the start of the buffer to the output. Your modification would work with a similar modification of the output.append line, but you would have to make sure your buffer isn't too small to hold all the input or you will get an error. Anyway, I think a better re-visiting of the code is:


I should really double check this before I post ... But I will edit it if I find something wrong. sorry about the poor memory before.
 
Dave Robbins
Ranch Hand
Posts: 131
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
I will ponder your revised code when it isn't dark:thirty and I don't have a beer in my hand
thanks for the tip, you got me over the hump

Dave
 
It is sorta covered in the JavaRanch Style Guide.
  • Post Reply Bookmark Topic Watch Topic
  • New Topic
Boost this thread!