This week's book giveaway is in the Kotlin forum.
We're giving away four copies of Kotlin in Action and have Dmitry Jemerov & Svetlana Isakova on-line!
See this thread for details.
Win a copy of Kotlin in Action this week in the Kotlin forum!
  • Post Reply Bookmark Topic Watch Topic
  • New Topic

Multi-threaded FTP Upload from JSE App  RSS feed

 
Kabron Kline
Greenhorn
Posts: 20
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
I am working on a solution that will provide multiple simultaneous file upload functionality. In order to facilitate this, I created a few classes using the FTPClient provided in the Apache Commons Net library as the underlying socket client. Note that I built my own abstraction called, "FTPClient" and composed it with the org.apache.commons.net.ftp.FTPClient class.

First is the "FileUploader" class which is used to invoke the upload process via the 'upload()' method (note: internal package names are shortened for brevity).

Second is the FileUploaderThread which implements Runnable and is responsible for polling files from a thread-safe queue and uploading them via calls to an FTPClient instance. Each FileUploaderThread instance gets a single FTPClient instance.

For a small number of files (e.g. 100), I can invoke the code via a JUnit test without any problems. If I use a larger file set which more closely reflects what will occur in our production environment (e.g. 5000+ files), the JVM will deadlock after a certain amount of time. Further, the same FTPClient upload code is being used in single threaded process without any problems.

I've tried a variety of things in an attempt to resolve my problem, but at this point I'm starting to run out of new ideas. My suspicion is that since the files I'm uploading are very small (less than 5 KB), there are two many socket connections occurring within a very short period of time.

I don't have much experience with multi-threading as it applies to socket connections. As such, any criticisms, recommendations, or suggestions are all welcomed!
 
Joe Ess
Bartender
Posts: 9429
12
Linux Mac OS X Windows
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
My suspicion is that since the files I'm uploading are very small (less than 5 KB), there are two many socket connections occurring within a very short period of time.


This is possible, though I would expect an exception to be thrown: TooManyOpenFiles
 
Kabron Kline
Greenhorn
Posts: 20
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Thanks for the reply! In my FTPClient class, all InputStream objects are closed in a finally statement after each upload. I validated via the debugger to make sure.
 
Joe Ess
Bartender
Posts: 9429
12
Linux Mac OS X Windows
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
You should also use lsof/Process Explorer (per the link) to make sure you have available sockets. Sockets are not returned to the available pool immediately.
 
Kabron Kline
Greenhorn
Posts: 20
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Thanks for calling out Process Explorer, I'll be using it to help troubleshoot. As a side note, when I change the for loop in the 'upload()' method of the FileUploader to reflect the snippet below, TimeoutException is caught after 1000+ file uploads.
 
  • Post Reply Bookmark Topic Watch Topic
  • New Topic
Boost this thread!