Hmmm... my guess is that it's some sort of memory leak in a native code library, which isn't part of the JVM heap, and won't necessarily throw an OutOfMemoryError since it's not managed by the JVM directly. I would suggest first, upgrade to the latest JDK if you haven't already; assorted past leaks have generally been fixed. Then, try to figure out which classes/methods are involved with the growing memory. Search Sun's
bug database for open bugs with the words "memory leak" plus the names of classes and methods you're using, to see if the problem has been reported already, and if there's any useful feedback. Try
testing your application carefully, watching memory as you perform various operations, to see if you can figure out which specific activities are responsible. It may also be useful to temporarily disable sections of code, to rule them out as possible culprits.
Lastly, it may be a good idea to invest in a good performance analysis tool like JProbe. However I don't really know if this will be any use if the problem is indeed in native code not managed by the JVM. Perhaps others who have used such tools more extensively can offer opinions here.
Good luck...