Win a copy of Practical SVG this week in the HTML/CSS/JavaScript forum!
  • Post Reply Bookmark Topic Watch Topic
  • New Topic

java.io.IOException:Too many files open

 
Sri Ram
Ranch Hand
Posts: 118
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
I am running java program involving 5 threads which runs continuously. I am writing the start and stop time of each thread in a log file. The code is given below.


The threads run for around 30500 times and then gives, too many files open error. I looked up in google, and found that a file can be open only for 2035 times and then it gives this error. But the threads here run for more than that much time. Can any one tell me why this error comes and whats the solution.
[ October 03, 2005: Message edited by: Ilja Preuss ]
 
Amol Fuke
Ranch Hand
Posts: 129
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Is this code in loop?

what is iCount++ ??
 
Sri Ram
Ranch Hand
Posts: 118
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Sorry Forget the count++. I just wanted to find the number of times the loop is executed.
 
Ilja Preuss
author
Sheriff
Posts: 14112
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Well, besides the fact that the synchronization on the local file variable doesn't serve any purpose, and that you don't need to flush a writer before you close it, I can't spot anything wrong in the code.

You might want to show us how this code is used...
 
Sri Ram
Ranch Hand
Posts: 118
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Solved. As it turned out the thread sleep time i have given is around 1 mill second. Hence the GC was not performed to the fullest ability. So when i tried to cound the number of references to the file using ProcessExplorer tool, viola there were 2035 files.

Thanx anyway for the help.
 
Tony Morris
Ranch Hand
Posts: 1608
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator

I can't spot anything wrong in the code.

I can
Weren't we discussing this just the other day on another thread?
What happens if the call to flush() throws an IOException?
*drip drip* something is leaking.
 
Ilja Preuss
author
Sheriff
Posts: 14112
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Originally posted by Tony Morris:

I can
Weren't we discussing this just the other day on another thread?
What happens if the call to flush() throws an IOException?
*drip drip* something is leaking.


Oops. I guess when I wrote that, I already had the refactored code without the flush in my mind...
 
Mr. C Lamont Gilbert
Ranch Hand
Posts: 1170
Eclipse IDE Hibernate Ubuntu
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
I am having trouble understanding why Sun felt the need to leave open file descriptors? I am seeing reports every now and again about people running into problems because of this. Plus, if you are running out of descriptors, why doesent the JVM close them? I think someone at Sun does not understand the role of the JVM. It manages memory, it does not manage any other resources.

Strange on their behalf.
 
Ilja Preuss
author
Sheriff
Posts: 14112
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Originally posted by Mr. C Lamont Gilbert:
I am having trouble understanding why Sun felt the need to leave open file descriptors?


The way I understand it, as long as a file is open, you *need* a file handle. Nothing specific to Java or Sun.


I am seeing reports every now and again about people running into problems because of this. Plus, if you are running out of descriptors, why doesent the JVM close them?


Which one should it "close"? How should it know which ones are still needed and which aren't?

I think someone at Sun does not understand the role of the JVM. It manages memory, it does not manage any other resources.


I guess they decided that the JVM shouldn't know about such OS specific things as file handles - it's the responsibility of the API implementation to know about it. I would have done it the same way.
 
Mr. C Lamont Gilbert
Ranch Hand
Posts: 1170
Eclipse IDE Hibernate Ubuntu
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Originally posted by Ilja Preuss:


I guess they decided that the JVM shouldn't know about such OS specific things as file handles - it's the responsibility of the API implementation to know about it. I would have done it the same way.


Show me the API that gives a reference to a filehandle You do know that inserting calls to System.gc() will hide this issue? You agree with that?
 
Maximilian Xavier Stocker
Ranch Hand
Posts: 381
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Originally posted by Mr. C Lamont Gilbert:


Show me the API that gives a reference to a filehandle You do know that inserting calls to System.gc() will hide this issue? You agree with that?


Honestly so what? This issue reminds me of the problem you can run into with too many open sockets.

In a nutshell if you open and close thousands of sockets you will run out of sockets and this is totally an OS issue and you can't so anything about it directly. All you can do is wait for the OS to do it's version of garbage collection on the sockets to release resources.

In either of these cases I think one has to take a second look at the design of your application and determine if it is really neccessary to tax the OS like this. I would bet that 99.9999% of the time it is not neccessary and a better design and flow would fix this.

Because back to my original point... OS's are not designed to create and destroy thousands of sockets per second.... and I think the same applies to files regarding the JVM. I certainly don't want to pay the GC penalty every time I close a file just because a few people have problems when they are dealing with thousands of file handles. This is exactly what something like GC is for and thus I really don't see it as a failing of the JVM that in this case you need to call it.

Finally I am not convinced that is the JVM that is the problem.. it could well be the OS itself is not releasing the resouces fast enough (just like the socket issue). In this case running GC gives the OS IO routines a pause to clean up.
 
Ilja Preuss
author
Sheriff
Posts: 14112
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Originally posted by Mr. C Lamont Gilbert:
Show me the API that gives a reference to a filehandle


FileInput/OutputStream.close(). That will release the file handle, won't it?

You do know that inserting calls to System.gc() will hide this issue? You agree with that?


Sorry, I don't understand the question.

Anyway, this should probably already have been moved to the IO forum earlier - let's see what the experts have to say...
 
Maximilian Xavier Stocker
Ranch Hand
Posts: 381
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
FileInput/OutputStream.close(). That will release the file handle, won't it?


Not directly no.

He is saying that calling gc does cause the handle to actually be released. I still say that it is not clear whether this means the JDVm is holding on to it or if this is an OS issue that GC treats the symptoms of.
 
Maximilian Xavier Stocker
Ranch Hand
Posts: 381
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
And I don't believe this thread should have been moved either because it is not really an IO issue in my mind but both a question of design and a discussion about resources in general and the JVM/OS.

I would reiterate though on the VM point that I can and have recreated these types of problems in other languages so the fact is it is not a VM issue.
 
Mr. C Lamont Gilbert
Ranch Hand
Posts: 1170
Eclipse IDE Hibernate Ubuntu
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Originally posted by Ilja Preuss:

FileInput/OutputStream.close(). That will release the file handle, won't it?


From what I read the answer is no, its sorta cached. But its only 2nd hand information and I don't want to throw myself on the sword for it


If the JVM holds the file handles then its buggy. If the OS holds the filehandles its the JVMs job to present this reality to the user in some form. If gc really does fix this, then its the JVMs fault and it needs to be fixed.
[ October 08, 2005: Message edited by: Mr. C Lamont Gilbert ]
 
Maximilian Xavier Stocker
Ranch Hand
Posts: 381
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Originally posted by Mr. C Lamont Gilbert:

If the JVM holds the file handles then its buggy. If the OS holds the filehandles its the JVMs job to present this reality to the user in some form. If gc really does fix this, then its the JVMs fault and it needs to be fixed.

[ October 08, 2005: Message edited by: Mr. C Lamont Gilbert ]


No it isn't Java's fault. It isn't a Java problem. It's an OS problem. That's what I am saying. The reason GC seems to work is not the neccessarily the reason you think. If you Thread.sleep you would see the same thing.

Why?

Because while in GC Java is doing stuff on the processor the OS has been relieved of the expensive ongoing flood of IO requests. This gives the OS a chance to deal with the backlog cleanup.

At the end of the day ANY program can only make requests of the OS to release held resources. But on most OS's the actions to release said resources are held as a lower priority then dealing with new requests. So when you flood the OS with requests for new sockets, file handles or whatnot it stops actually getting rid of the old ones until it has a break.. which your app never gives it and down it goes.

There is no way to fix this in Java because as I said before you can have the same problem in C or VB or whatever you want and you can't fix it from there either other than to not do this in the first place or occaisonally pause and let the OS do it's GC to actually release the resources you have discarded.
 
Mr. C Lamont Gilbert
Ranch Hand
Posts: 1170
Eclipse IDE Hibernate Ubuntu
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
OK. I will keep that in mind.
 
Ilja Preuss
author
Sheriff
Posts: 14112
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Originally posted by Maximilian Stocker:
And I don't believe this thread should have been moved either because it is not really an IO issue in my mind but both a question of design and a discussion about resources in general and the JVM/OS.


Looks to me like the main question is what part of the system is/should be responsible for the management of file handles, which I hoped an IO expert could bring light into. That's why I moved it.

Of course as a Sheriff I'm only *nearly* god, so you are kind of free to disagree...
 
Ilja Preuss
author
Sheriff
Posts: 14112
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Originally posted by Mr. C Lamont Gilbert:
From what I read the answer is no, its sorta cached. But its only 2nd hand information and I don't want to throw myself on the sword for it


Well, the JavaDoc to FileOutputStream.close says


Closes this file output stream and releases any system resources associated with this stream.


Similar for FileInputStream. Writers and Readers delegate to their streams.

So if the JavaDoc is correct (the implementation is native), after a close operation it would be up to the OS to actually make the file handle available for future use.
 
  • Post Reply Bookmark Topic Watch Topic
  • New Topic
Boost this thread!