If you are using a 32-bit JVM, the max heap size is usually around 1.5GB on Windows and around 2GB(not sure on this one) on Linux. The exact amount varies depending on the exact version of the OS (including patch levels), and on other variables such as the amount of permgen space allocated. If you are using a 64-bit JVM, the heap is limited only to the amount of RAM plus swap space that you have.
If your app is really that memory hungry,
you should run it in a separate JVM.