There are different things
you should consider.
1) check whether there are any concurrent applications running i.e. Database server, MQ server, any other application consuming memory and CPU.
2) Since you say that your application is showing outofmemory, you will need to revisit your application API and check the object creation and scope properly. You can use any profiling tool such as Jprobe from quest software Or optimizeIt software. I have personally used JProbe and it gives an excellent insight on the CPU, memory and object state.
3) Perform performance and load testing on the application. Using automated testing software, load test the application to find out at what stage the application shows outofmemory. You should be able to pinpoint the functionality which is having bottleneck.