• Post Reply Bookmark Topic Watch Topic
  • New Topic

Error "too many files open" despite setting the max thread count using Thread Pool.  RSS feed

 
Monica Shiralkar
Ranch Hand
Posts: 922
2
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
I created a thread pool and set the maximum number of threads as 50 only.Despite this I am getting error "too many files open".That means more than 50 threads are working at a time which should not happen.



What I am trying to do:

Each thread is writing an output file for each input file in dir1 directory. Since these should not be too many open files at a time I am setting maximum thread to 50 but still I get the message. Is pooling not working.Please advice.

thanks
 
sai rama krishna
Ranch Hand
Posts: 536
1
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
How are you invoking this code?
 
Ulf Dittmer
Rancher
Posts: 42972
73
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
How do you figure the correlation between threads and open file handles? While a file will certainly use one, lots of other things will as well, not all of them necessarily related to your code.

The usual solution is to bump the max file limit way up. It frequently is set to 1024, whereas modern servers can handle multiples of that without problems.
 
Monica Shiralkar
Ranch Hand
Posts: 922
2
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
There is a folder with 1 million files. I am reading these files (Not through thread but sequentially), doing my processing through 1 thread for each file and then each thread writes to a different output file. I understand that reading 1 million files should not happen by threading.So I am reading them sequentially but processing can happen through threads thats what I am doing. Now after processing each thread will write an output file for this it will have to open file and then write. I get message "too many open files". But surprising thing is that I have set the maximum thread count as 50 which is less than 1024 they why do I get this message. Please advice on what can be possible solution.

thanks
 
Paul Clapham
Sheriff
Posts: 22835
43
Eclipse IDE Firefox Browser MySQL Database
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
The error message suggests that you aren't closing the files after you read them, or write them, or whatever you are doing. And there's nowhere in your posted code which shows files being closed (or opened, for that matter). Probably that takes place in the CDRDecoder class? Anyway you should check that you're closing files after you're finished using them.
 
Monica Shiralkar
Ranch Hand
Posts: 922
2
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Thanks.

Each thread is creating a file and then opening it.When processing of a thread is complete, the file is closed. At any point of time I expect maximum 50 files to be open but still it is giving the message.

I am allowing 50 threads to run at a time.

Each thread does the below:

Step1: Create a file
Step 2: Do the processing.
Step 3: Close the file


Despite allowing 50 as the maximum number of threads running at a time, I get a message "too many open files". What can be a solution to this. Please advice.
 
Ulf Dittmer
Rancher
Posts: 42972
73
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
I hadn't noticed before that you were using Windows; the 1024 I mentioned is a number for Unix-like OSes.

That means more than 50 threads are working at a time which should not happen.

What it actually means is that more files are being opened than are being closed, and that the difference is beyond some limit that Windows has (and I have no idea what that limit is). I would imagine that the files being created by processes other than this one also count towards this limit, so make sure nothing I/O intensive is going on in the background.

I'm with Paul in thinking that the files may not get closed properly, or at least not on all possibly code paths. Can you post relevant code excerpts that deal with I/O, opening and closing?
 
Monica Shiralkar
Ranch Hand
Posts: 922
2
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Thanks.. below is the relavent code:

The number of threads running at a time I have limited to 50.

 
Ulf Dittmer
Rancher
Posts: 42972
73
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
That code doesn't look like it would even compile -what with the "else" without an "if"- but I notice it doesn't close byteStream (whatever that is) and bw if there's an exception - are there exceptions?
 
Monica Shiralkar
Ranch Hand
Posts: 922
2
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Actually the whole code is too big.I had sent you parts of the relavent code.

The line at which it throws exception is :



The exception is Exception::java.io.IOException: Too many open files
java.io.IOException: Too many open files


I thought that becuase of the below code it will create 50 threads at a time but it gives too many open files:




Below is the code which is closigng the file:


bw.write(content + "\n");
// bw.write("\n'");


//System.out.println("outputCount"+outputCount);

bw.close();

byteStream.close();


The root cause I feel is that even though I have controlled the number of threads, the number of files getting opened at any point of time is becoming too high.What can be a solution.I am using UNIX

 
Ulf Dittmer
Rancher
Posts: 42972
73
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
You haven't said if there are exceptions.

If you are using Unix/Linux, then my earlier comment does apply. Check how many files are allowed (ulimit -a), and if it's 1024, bump it up via "ulimit -Hn 4096" or some other value. It's also useful to check out which files are open via "lsof". A "file" in Unix-ish OSes are not only what you'd normally think of as files, it can also be network connections and various other things.
 
Monica Shiralkar
Ranch Hand
Posts: 922
2
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
thanks.limit is 1024. I will do these.What about the changes in code ? Is approach of starting and closing files in threads and keeping limited number of threads correct?
 
  • Post Reply Bookmark Topic Watch Topic
  • New Topic
Boost this thread!