You need to monitor garbage collection and run through the application with a fine toothed comb. Try setting pointers to objects that you no longer use to null and avoid static variables as possible.
Also, you can request a garbage collection (often after a large memory-intensive process) but there's no guarentee it will work and
java won't let go of memory that it thinks it needs.
My tendency would be to think this is a programming problem, although configuration is always possible. The easy check is try the same MDB config with dummy code that does nothing. It should give you an idea either way.