especially when the server is left idle for sometime.
Sounds like you are running Windows. If you leave an app idle long enough, Windows will place the whole thing out in the pagefile. Then when you need it again, Windows has to page the whole thing back in, which can be really time consuming because it doesn't do it in bulk but in pieces.
Another possibility is that is the app server is running from a disk other than the one holding the OS, the machine has powered down the disk and you have to wait for the disk to spin back up to swap in pieces of code. I've experienced this on my desktop running both Windows and Linux.