• Post Reply Bookmark Topic Watch Topic
  • New Topic

Download only a bit of a file and stop.  RSS feed

 
Krystian Gor
Ranch Hand
Posts: 30
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
I'm using a simple construction as this one:The output I get is:
But when I watch it on wireshark I can see whole page code being downloaded until </HTML>. How do I break connection after <body>? How do I read stream by fetching only x bytes?
 
Jeff Verdegan
Bartender
Posts: 6109
6
Android IntelliJ IDE Java
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
You can't force the underlying network to not send more data than you asked for. Closing the connection instead of just the InputStream might cut off the flow of bytes sooner, but it's definitely not guaranteed how much sooner, if at all.

It shouldn't matter though. Your program isn't wasting CPU pulling in stuff you don't ask for. It's being buffered lower in the network stack. And if that network stack and the software and firmware on the various routers and switches between you and the other host is written reasonably well, it won't be choking up your network with unasked-for bytes if there's other stuff explicitly asking for transport.
 
Krystian Gor
Ranch Hand
Posts: 30
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Now I see. I debug line by line and when my program executes conn.getInputStream(); Wireshark instantly shows full HTML code of this site (from <html> to </html>). I didn't even start reading this stream with readline() nor with read(). Wow, I didn't know it works like this. I thought data is being downloaded part by part inside my while loop.
This program IS wasting bandwidth as I don't need <body>...</body>(100kB of data). I only want <head> ... </head>(first 1kB of data).

EDIT:
I will try to modify HTTP request to make response contain only a selected part of data (Range property)
 
Jeff Verdegan
Bartender
Posts: 6109
6
Android IntelliJ IDE Java
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Krystian Gor wrote:Wow, I didn't know it works like this.


It doesn't necessarily work that way all the time. But if you open an InputStream, that means you're establishing a connection and probably plan on downloading some data. It's not that surprising that it would pre-buffer some data for you, so that when you do ask for it, it's there. In fact, I wouldn't be surprised if the lower level TCP protocol has to send some bytes anyway--like a connection ACK or something--and, since it's going through the overhead of sending one or more frames anyway, it decides to piggyback some session- or application-level data on it.

If no data was ever put on the wire until the other end received your explicit request for it, the internet would be a much slower place.
 
Jeff Verdegan
Bartender
Posts: 6109
6
Android IntelliJ IDE Java
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Krystian Gor wrote:
This program IS wasting bandwidth as I don't need <body>...</body>(100kB of data). I only want <head> ... </head>(first 1kB of data).


I didn't say it isn't wasting any I said it's probably not wasting enough to damage other transmissions that actually need the data. And 100kB isn't all that much on today's networks. Plus, contrary to my other guesses, it probably isn't the case that the lowest level is just bringing data along in anticipation. That data could be there if it wasn't explicitly written by the server. Since you're connecting to a web server and asking for a page, the most common use case is that you're going to want the whole page, and you're not going to want to wait for it, so it's perfectly reasonable--even desirable--for the server to start writing as much as it can as fast as it can, and let the network and the endpoints' TCP/IP stacks buffer as much as possible so it's ready as soon as you ask for it.
 
Krystian Gor
Ranch Hand
Posts: 30
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Thank you.
 
It is sorta covered in the JavaRanch Style Guide.
  • Post Reply Bookmark Topic Watch Topic
  • New Topic
Boost this thread!