• Post Reply
  • Bookmark Topic Watch Topic
  • New Topic

Gracefully handling IOException: too many open files

 
Pete Nelson
Ranch Hand
Posts: 147
Debian Eclipse IDE Tomcat Server
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
I am writing a program to analyze files in a directory. Basically, the java program creates a CSV file for loading into a database, where I can check md5 sums to find duplicate files.

The problem I am running into is a common one: IOException due to "Too many open files". I see a lot of answers that involve trying to change the OS, rather than resolve the issue in the Java code ... but I'd really like to find a way to mitigate this in the code, rather than forcing changes to the system.

I have an ArrayDeque (fileQueue) that gets populated with the canonical path of the files to process. The exception is occurring when I call my processFiles() method:



I'm setting my File object (f) to null in a finally block ... I'm doing the same to my custom FileInfo object (a wrapper to format the file information). I've reduced THREAD_COUNT from 10 to 2, and still get the IOException. I don't see where files are "staying open" here. In a desperate attempt, I even tried sleeping the thread, to see if that would allow time for other threads to close files, but still no luck.

Does anyone know how to handle this more gracefully, or see what I am doing wrong?

(please, don't tell me about ulimit and files-max ... I understand what that should do, but don't believe I should have to do that!)

Any help on making the Java program more graceful would be greatly appreciated (I KNOW there's a way to make this program behave!)

 
Rob Spoor
Sheriff
Pie
Posts: 20659
64
Chrome Eclipse IDE Java Windows
  • Likes 1
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Pete Nelson wrote:I'm setting my File object (f) to null in a finally block ... I'm doing the same to my custom FileInfo object (a wrapper to format the file information).

That doesn't necessarily close the resources. You need to explicitly close() all streams when you're done with them, preferably in a finally block.
 
Pete Nelson
Ranch Hand
Posts: 147
Debian Eclipse IDE Tomcat Server
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Good tip, Rob!!! Thank you

Sure enough, I looked inside my FileInfo class, and found this -

The FileInputStream serves no other purpose in my class than to generate the MD5 sum. By explicitly calling close(), and wrapping in a try/catch/finally block, all is as it should be!
 
  • Post Reply
  • Bookmark Topic Watch Topic
  • New Topic