There are a lot of things that can cause out of memory errors. If you notice that a particular operation, like a database query or file load, causes it, you may be trying to load too much data in the virtual machine. You can either increase the VM's size or process the data in smaller pieces.
If the problem is more intermittent, it may be caused by not freeing up resources correctly. Any file, socket, database connection or other resource must be freed by invoking close(). This must be done in the finally block of a try-catch-finally structure to make sure close() is invoked.
If the cause is not obvious, you are probably accumulating data somewhere and never freeing it. HashMap/Table can have this problem if your objects don't implement hashcode correctly. The JVM has some tools to help you monitor the VM:
Monitoring and Managing Java SE 6 Applications.