There are a lot of things that can cause out of memory errors. If you notice that a particular operation, like a database query or file load, causes it, you may be trying to load too much data in the virtual machine. You can either increase the VM's heap memory or process the data in smaller pieces. To increase the heap memory, use the -Xmx command-line switch
described here
If the problem is more intermittent, it may be caused by not freeing up resources correctly. Any file, socket, database connection or other resource must be freed by invoking close(). This must be done in the finally block of a try-catch-finally structure to make sure close() is invoked. You can use tools like
lsof on Linux/Unix OS's and
Process Explorer on Windows to monitor open file descriptors for a process. If the number of descriptors keeps increasing for your server process, this is your problem.
If you are working with a Swing program, a common cause of memory leaks is adding listeners to controls and never releasing them.
If the cause is not obvious, you are probably accumulating data somewhere and never freeing it. HashMap/Table can have this problem if your objects don't implement hashcode correctly. The JVM has some tools to help you monitor the VM:
Monitoring and Managing Java SE 6 Applications
Handling Memory Leaks is another good article to help understanding and fixing memory problems
