• Post Reply Bookmark Topic Watch Topic
  • New Topic

Looking for multi-threaded java downloader  RSS feed

 
Justin Thomas
Ranch Hand
Posts: 62
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Downloading a file is easy, but downloading it as a multi-threaded download (like most download accelerators) is a whole different story....

I am looking for such a downloader (preferably with simple API). So far I found:


Yet nothing seems to be conclusive. What multi-threading API would YOU suggest?
 
Stephan van Hulst
Saloon Keeper
Posts: 7993
143
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Don't most websites limit the amount of connections you can make anyway?
 
Justin Thomas
Ranch Hand
Posts: 62
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Stephan van Hulst wrote:Don't most websites limit the amount of connections you can make anyway?

Not 'good' websites. I compiled the jdownman on my own (added missing lib dependencies, and it all compiled nicely).
I saw the Download Test and SimpleDownloadTest in test pkg however they do not specify how to make it multi-threaded. Though I am sure it supports it, because I saw many lines of code like "chunkDownload" and "beginRange" and "endRange". So I guess it's there somewhere....
 
Justin Thomas
Ranch Hand
Posts: 62
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Well, there is an obvious issue of authorization, lack for ftp support and the fact that I still haven't figured out how to get it running in multi-threaded mode (though I made some progress)
 
Justin Thomas
Ranch Hand
Posts: 62
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
In jdownman project, at DownloadImpl.size() I did this:


Now it kind-of-works*. It splits the download into parts and downloads each part concurrently (+supposedly according to the owner of the project, it keeps the bandwith distribution fair, so no chunk is left alone, however the only thing I saw from the code was a lot of playing with the buffer sizes) Yay for me :-) I actually made someone else's tool working :D <- My first time.

kind-of-works*
1. Changing HeadMethod to GetMethod means that besides downloading file into chunks, it also downloads one single same file (silently in the background, almost unnoticably). So it doubles the work.
2. Wtih jdownman there is a bug of setting the download size, because DownloadManagerImpl.download(params...) takes long length and based on length it decides how to split the file, however the DownloadImpl is actually getting the real size of the file - so the long length param is actually unnecessary AND should be replaced by automated function that fetches size (because it already exists, but in different class).
3. Any possible output/verbosity output/debug output is done through DownloadListeners - theoretically - it's a good programmer's approach. Practically - without any javadoc/wiki/testcase/anything for the project, writing working Listener (and it's a one of those more super uber nifty hardcore listeners from my first glance at the class) is beyond the scope of my skills. So the only thing I did was some System.out.println in various places so I know at least a bit about what's going on inside...

So with 1-2 days of working I actually might make the jdownman to a state where it is being usable and works. I thought of contributing my changes, but then again, the project is left for dead long ago, so I guess the owner won't commit my changes anyways....

--------
ps: Can anyone suggest some Listeners tutorial? I can barely write a simplest listener that works, and my idea about them is as vague as with US football (I live in EU). The tutorials I found on google presume I know certain `stuff` and I couldn't find any tutorial that would really begin at the beginning.
 
  • Post Reply Bookmark Topic Watch Topic
  • New Topic
Boost this thread!