I have got OOM exception with really heavy load
test several hundreds users..
Here is what I see in the heap dump, see below:
It takes 50% of memory
It looks like it comes to oracle connections and shared pool at the lowest level...
Any ideas?
461,649,576 (57%) [32] 4 array of java/lang/Object 0x102e8e40
461,627,888 (57%) [120] 14 com/ibm/ws/webcontainer/srt/SRTServletRequest 0x3d4a35a0
461,620,848 (57%) [32] 4 com/ibm/ws/webcontainer/srt/SRTConnectionContext 0x3d4a3618
461,620,816 (57%) [40] 5 com/ibm/ws/webcontainer/srp/SRPConnection 0x20b32190
461,617,808 (57%) [88] 12 com/ibm/ws/webcontainer/http/HttpConnection 0x20e58c78
460,944,752 (57%) [32] 4 com/ibm/ws/webcontainer/oselistener/OSEListenerDispatcher 0x20b321b8
460,944,608 (57%) [24] 2 com/ibm/ws/webcontainer/srp/ServletRequestProcessor 0x20da4f70
460,944,408 (57%) [88] 16 com/ibm/ws/webcontainer/WebContainer 0x18b15b78
460,875,072 (57%) [80] 5 com/ibm/ws/util/ThreadPool 0x1a854378
460,541,952 (57%) [24] 2 array of com/ibm/ws/util/ThreadPoolListener 0x1a849d20
460,541,912 (57%) [40] 5 com/ibm/ws/runtime/component/ThreadPoolMgrImpl 0x13465670
460,536,560 (57%) [48] 1 java/util/HashMap 0x1323c9b0
460,536,512 (57%) [80] 5 array of java/util/HashMap$Entry 0x1ab26718
460,536,272 (57%) [32] 3 java/util/HashMap$Entry 0x117281a0
460,536,112 (57%) [80] 5 com/ibm/ws/util/ThreadPool 0x11794028
460,535,656 (57%) [32] 4 com/ibm/ws/runtime/component/ThreadPoolMgrImpl$ThreadPoolMonitorAdapter 0x11728180
460,535,600 (57%) [88] 8 com/ibm/ws/runtime/component/ThreadMonitorImpl 0x13307d00
460,534,880 (57%) [56] 5 com/ibm/ws/runtime/component/ServerCollaborator 0x13211a60
460,529,584 (57%) [24] 3 javax/management/modelmbean/RequiredModelMBean 0x11c9c688
460,484,880 (57%) [24] 2 com/tivoli/jmx/modelmbean/MMBNotificationBroadcasterSupport 0x11c9c658
460,484,392 (57%) [24] 2 com/tivoli/jmx/modelmbean/MMBNotificationBroadcaster 0x11c9c640
460,484,368 (57%) [16] 1 javax/management/NotificationBroadcasterSupport 0x11c9c630
460,484,352 (57%) [16] 1 java/util/HashSet 0x11c9c620
460,484,336 (57%) [48] 3 java/util/HashMap 0x11c9c5f0
460,484,256 (57%) [80] 1 array of java/util/HashMap$Entry 0x11c9c5a0
460,484,176 (57%) [32] 2 java/util/HashMap$Entry 0x11ca0338
460,484,144 (57%) [24] 2 com/tivoli/jmx/CollectableListener 0x11ca0368
460,484,120 (57%) [24] 3 com/tivoli/jmx/ListenerRepository$ListenerProxy 0x11ca0388
460,484,080 (57%) [24] 2 com/tivoli/jmx/ListenerRepository 0x10e73a20
460,483,320 (57%) [24] 2 com/tivoli/jmx/MapAdapter 0x109a1488
460,483,296 (57%) [48] 2 java/util/HashMap 0x109a1458
460,483,232 (57%) [1,040] 96 array of java/util/HashMap$Entry 0x196f1f60
459,980,040 (56%) [32] 2 java/util/HashMap$Entry 0x14c55020
459,978,280 (56%) [32] 2 com/tivoli/jmx/GenericMBeanSynch 0x14c55040
459,978,248 (56%) [24] 3 javax/management/modelmbean/RequiredModelMBean 0x14c55dc8
459,974,496 (56%) [24] 3 com/tivoli/jmx/modelmbean/DynamicModelMBeanSupport 0x14c55b80
459,974,472 (56%) [72] 10 com/ibm/ws/webservices/component/WSServerImpl 0x116d8290
459,965,000 (56%) [144] 22 com/ibm/ws/runtime/component/EJBContainerImpl 0x1179aee0
459,942,720 (56%) [48] 1 java/util/Hashtable 0x116ed9e0
459,942,672 (56%) [64] 6 array of java/util/Hashtable$Entry 0x116ed9a0
459,939,856 (56%) [32] 2 java/util/Hashtable$Entry 0x180d02e0
459,939,784 (56%) [56] 10 com/ibm/ejs/csi/EJBConfigDataImpl 0x180d0300
459,411,904 (56%) [432] 66 com/ibm/ejs/container/BeanMetaData 0x180d0130
459,382,600 (56%) [32] 2 com/ibm/ws/naming/java/javaNameSpaceImpl 0x180d1910
459,382,440 (56%) [32] 2 com/ibm/ws/naming/ipbase/NameSpace 0x180d18f0
459,376,144 (56%) [48] 1 java/util/HashMap 0x180d1840
459,376,096 (56%) [80] 4 array of java/util/HashMap$Entry 0x180d17f0
459,372,968 (56%) [32] 2 java/util/HashMap$Entry 0x180e65f8
459,372,752 (56%) [80] 9 com/ibm/ws/naming/java/javaURLContextImpl 0x180e6898
459,366,360 (56%) [408] 39 com/ibm/CORBA/iiop/ORB 0x120c3130
457,769,480 (56%) [88] 5 com/ibm/rmi/iiop/ORB$3 0x1025f8c0
457,769,312 (56%) [144] 17 com/ibm/ws/bootstrap/ExtClassLoader 0x1006dbb0
456,010,192 (56%) [32] 1 java/util/Vector 0x100d11a8
456,010,160 (56%) [40,976] 6,832 array of java/lang/Object 0x12766620
433,039,728 (53%) [312] 5 class com/ibm/ejs/ras/Tr 0x1005fe08
432,902,712 (53%) [32] 4 com/ibm/ejs/ras/ComponentManager 0x12f68b70
432,671,200 (53%) [48] 1 java/util/Hashtable 0x12f68a88
432,671,152 (53%) [64] 3 array of java/util/Hashtable$Entry 0x12f68a48
432,364,448 (53%) [32] 3 java/util/Hashtable$Entry 0x15f69350
432,364,056 (53%) [32] 1 java/util/Vector 0x15f693a8
432,364,024 (53%) [56] 2 array of java/lang/Object 0x15f69370
432,363,968 (53%) [72] 6 com/ibm/ejs/util/cache/Cache 0x15f68d00
432,330,040 (53%) [16,464] 4,111 array of com/ibm/ejs/util/cache/Bucket 0x15f64c68
431,918,776 (53%) [40] 1 com/ibm/ejs/util/cache/Bucket 0x15f7b200
431,918,736 (53%) [56] 1 array of java/lang/Object 0x15f7b1c8
431,918,680 (53%) [40] 2 com/ibm/ejs/util/cache/Element 0x18106a10
431,918,608 (53%) [40] 2 com/ibm/ejs/container/EJSWrapperCommon 0x18106c58
431,918,520 (53%) [48] 8 com/mycompany/na/awd/awdapplications/bl/ejb/EJSLocalStatelessAWDAppsGTPSessionBLHome_549a10df 0x18106a38
431,623,096 (53%) [48] 6 com/ibm/ejs/container/BeanId 0x18107648
431,622,576 (53%) [40] 6 com/ibm/ejs/container/HomeOfHomes 0x15fe3178
431,622,520 (53%) [24] 1 com/ibm/ejs/util/FastHashtable 0x15fe3160
431,622,496 (53%) [8,232] 2,053 array of com/ibm/ejs/util/Bucket 0x15fe1138
431,563,736 (53%) [24] 2 com/ibm/ejs/util/Bucket 0x15fe0c88
431,563,712 (53%) [32] 3 com/ibm/ejs/util/Element 0x1810c9a0
431,563,440 (53%) [32] 4 com/ibm/ejs/container/HomeOfHomes$HomeRecord 0x1810c9c0
431,526,144 (53%) [144] 16 com/mycompany/na/awd/awdapplications/bl/ejb/EJSStatelessAWDEngineEventDAOBLHomeBean_9b105deb 0x1810a128
431,525,104 (53%) [56] 3 com/ibm/ejs/csi/PoolImplThreadSafe 0x1810d1c0
431,523,032 (53%) [32] 2 com/ibm/ejs/csi/PoolManagerImpl 0x15f3a1a8
431,522,912 (53%) [56] 6 array of com/ibm/ejs/csi/PoolImplBase 0x15f3a170
431,513,240 (53%) [56] 3 com/ibm/ejs/csi/PoolImplThreadSafe 0x17b28d80
431,510,680 (53%) [144] 15 com/ibm/websphere/startupservice/EJSStatelessAWDStartUpHomeBean_de54ea74 0x17b27bd8
431,508,864 (53%) [432] 53 com/ibm/ejs/container/BeanMetaData 0x174d7c28
431,393,880 (53%) [32] 2 com/ibm/ws/naming/java/javaNameSpaceImpl 0x174d5790
431,393,848 (53%) [32] 2 com/ibm/ws/naming/ipbase/NameSpace 0x174d5770
431,381,688 (53%) [48] 1 java/util/HashMap 0x174d5710
431,381,640 (53%) [80] 3 array of java/util/HashMap$Entry 0x174d56c0
431,377,528 (53%) [32] 2 java/util/HashMap$Entry 0x17c3feb8
431,377,312 (53%) [80] 9 com/ibm/ws/naming/java/javaURLContextImpl 0x17c3c768
431,376,352 (53%) [80] 9 com/ibm/ws/naming/java/javaURLContextImpl 0x17c3d470
431,375,464 (53%) [80] 9 com/ibm/ws/naming/java/javaURLContextRoot 0x174d8f70
431,374,776 (53%) [48] 1 java/util/Hashtable 0x174d8cd0
431,374,728 (53%) [64] 5 array of java/util/Hashtable$Entry 0x174d8c90
222,279,192 (27%) [32] 2 java/util/Hashtable$Entry 0x17a0d788
222,279,080 (27%) [48] 5 com/ibm/ws/rsadapter/jdbc/WSJdbcDataSource 0x17a0d8c8
222,212,216 (27%) [120] 10 com/ibm/ws/rsadapter/spi/WSManagedConnectionFactoryImpl 0x179c8a80
222,207,168 (27%) [48] 1 java/util/HashMap 0x179bb1a0
222,207,120 (27%) [528] 2 array of java/util/HashMap$Entry 0x179baf90
222,206,560 (27%) [32] 2 java/util/HashMap$Entry 0x25439b38
222,205,504 (27%) [144] 13 com/ibm/ejs/j2c/ConnectionManager 0x251dc108
222,142,040 (27%) [352] 26 com/ibm/ejs/j2c/poolmanager/PoolManager 0x179d76a8
187,497,208 (23%) [816] 200 array of com/ibm/ejs/j2c/poolmanager/SharedPool 0x179eec38
24,284,768 (3%) [64] 4 com/ibm/ejs/j2c/poolmanager/SharedPool 0x179ee600
20,331,152 (2%) [64] 4 com/ibm/ejs/j2c/poolmanager/SharedPool 0x179eb320
20,324,056 (2%) [64] 4 com/ibm/ejs/j2c/poolmanager/SharedPool 0x179ee3b0