Win a copy of Murach's Python Programming this week in the Jython/Python forum!
  • Post Reply Bookmark Topic Watch Topic
  • New Topic

Download large data from server (1 TB data)  RSS feed

 
Nirav Solanki
Greenhorn
Posts: 3
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
How can i download large amount of data (near 1 or 2 TB) from the server using any of the protocols or from web or with desktop application?
 
Tim Moores
Saloon Keeper
Posts: 3511
77
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
That's a lot of data - what bandwidth does the smallest pipe between both machines have? Use that to calculate how long it will take at least. And you can't assume that you will always get that full bandwidth, so it'll take even longer.

You'll need a tool that can restart downloads in the middle, since for such long download times there's a good chance of it getting interrupted at some point.
 
Nirav Solanki
Greenhorn
Posts: 3
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Tim Moores wrote:That's a lot of data - what bandwidth does the smallest pipe between both machines have? Use that to calculate how long it will take at least. And you can't assume that you will always get that full bandwidth, so it'll take even longer.

You'll need a tool that can restart downloads in the middle, since for such long download times there's a good chance of it getting interrupted at some point.


Thanks Tim for the revert. But what I am looking at is, for a solution through which I can download data upto 1TB from server using client/server app which can communicate over TCP/IP or SFTP protocol.
Would like to know how to code through which I should be able transfer data of a file from server which is of 15-20 GB.
 
Henry Wong
author
Sheriff
Posts: 22818
119
C++ Chrome Eclipse IDE Firefox Browser Java jQuery Linux VI Editor Windows
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Nirav Solanki wrote:
Thanks Tim for the revert. But what I am looking at is, for a solution through which I can download data upto 1TB from server using client/server app which can communicate over TCP/IP or SFTP protocol.


First, the SFTP protocol is a layer on top of the TCP/IP protocol -- along with a few other layers too.  Arguably, it would be difficult to find any file transfer protocol that does not use TCP/IP. This protocol is simply the standard on the internet, and it would be silly to not use it.

Tim point that the protocol should be restartable is a very good one. And SFTP is such a protocol. After you confirm that your server has the SFTP service setup, then simply download and install one of the many free clients that support it. Just make sure that the client supports the restart feature though.

Henry
 
Nirav Solanki
Greenhorn
Posts: 3
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Thanks for your reply guys.
I think based on the observation given by both of you  it's better to use SFTP and for download client can use FileZilla, WinSCP or any other application
 
Dwayne Barsotta
Ranch Hand
Posts: 83
1
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
I'd like to add a related question that lead to an answer, day you write and application that starts a down load, a file that sixer you know there is going to be an error of some type causing the download to fail (pause). Could it be feasible that in the exception section of this error, you application calls a method, maybe even the original download method (recursion), but in a new thread. The first thread stopped from the error, but the second thread gets a start point (you need to plan that in code), and starts the download from the failed part.  This procedure continues until file is completely downloaded, then you write code that merges the downloaded sections.  Of course yours have to add some sort of marker so you know when I file failed and started. 

Another idea would be to divide the original file into manageable sizes, encapsulate them and send them.  When all are sent your software reassembles the original file.


Do not take these as options, I am completely new, this are just thoughts I had with no educated background.

But for those in the know, how far out am I?
 
With a little knowledge, a cast iron skillet is non-stick and lasts a lifetime.
  • Post Reply Bookmark Topic Watch Topic
  • New Topic
Boost this thread!