Is this an application you have written? Without the source you will not be able to do much about a resource leak. What does the application do? Is it processing files or reading from a DB? If those resources are not properly closed they may cause leaks. In the try-catch-finally block surrounding such interactions, make sure the files and db connections are closed in the finally block. There are tools like JConsole, which can track class instances and memory usage and help you track down resource leaks.
What OS are you running on ? If you are on Linux or Solaris, try using Auptyma's Java Application Monitor. You don't need access to your application source code, and you can use it to take multiple snapshots of your system and compare them to see what objects/instances/JNI Handles etc have grown, and where they are reachable from.
<a href="http://www.auptyma.com" target="_blank" rel="nofollow">The Peak of Performance</a>
Have you tried the "poor man's troubleshooter" namely code traces. If you know what handle is increasing it should be quite easy to track down the problem. [ December 31, 2005: Message edited by: uj johansson ]
The code wasn't written by me, I even had to consider code which was not available. The problem seemed to be in the try-catch-finally as Joe Ess suggested. I was not sure this was the problem as I found the faulty pattern in many places and kept correcting it. Until suddenly the problem disappeared.
And the error itself was due to an aleatory appearance of a http connection exception due to a rather small "sun.net.client.defaultConnectTimeout". After closing the connections in finally and empirically chosing a bigger defaultConnectTimeout the pb. disappeared. Thanks, Cristian
PS: Sorry for late answer, this was solved quite a while ago.