I have a program that uses RMI to connect two servers.
The program is running on linux (kernel 2.6.9-42.0.3.EL) with java 1.6.0_13-b03.
After a while, while the two servers are up and communicating i get an exception "Too many open files" on one of the machines.
It seems that for a while FDs are open and cleaned correctly, but after a while they just start piling up.
There are many open socket file descriptors that i see by running the command "lsof" -
"java 3588 root 1005u sock 0,4 2213683 can't identify protocol"
ulimit is set correctly to -n -> 65535, -u -> 49148, -c -> unlimited. The total number of open FDs is much less that the limit (about 3700).
On netstat i can see only one process that uses RMI and two or none in CLOSE_WAIT state.
Every stream that i open is closed correctly.
What could be causing the too many open files exception?