I see that it's been a long time since this was posted, but just incase anyone is looking for a problem I found with WebSphere v5.0.2 and came across this by google search I'll go ahead and post.
I found that WebSphere v5.0.2 running on AIX 5.1 had a memory leak in it's ibmorb.jar that would never release entries that it stored in an internal cache. IBM had to do a hot fix for my client and as far as I know this was not made publicly available. It will probably be fixed by IBM in their WAS 5.1 release. The scenario where this arises is when one application makes a remote
EJB call to another application. The IBM ibmorb.jar does not release references to the EJB stubs and will continue to amass references until your server instance eventually encounters an OutOfMemoryError.
Here is the explanation from IBM:
1.The Problem analysis for the Defect 74270 is as below:
The identityHashTable was holding references to objects (stubs) , thus
preventing Garbage collection of objects. This happens especially
when the EJB from a different jar file is being accessed by a
servlet in a different ear file.
2.The Problem resolution for the same as below:
Replaced identityHashtables with weakHashMaps.
-Ed