Since this means you would have 2 different JVM's I would think there is no 'java way' of doing this.
First method that pops in my mind is using a "LOCK" file, on which your program holds on to while it's running. If it's not free when your program starts, it will terminate itself. But I'm pretty shure there are better ways for this.
Another possible solution is to use a socket instead of a file.
Pro: the socket gets closed automatically when the program exits, even when the VM crashes or something.
Con: the socket could be used by something else.
The soul is dyed the color of its thoughts. Think only on those things that are in line with your principles and can bear the light of day. The content of your character is your choice. Day by day, what you do is who you become. Your integrity is your destiny - it is the light that guides your way. - Heraclitus
Your program could listen on a port and reply back to a second instance that attempts to start. This would work across JVM's, but would make the zonealarm/kerio, etc go nuts-o. This is more reliable than the existance of a lock file because if you have a crash, your lock file remains and then you are SOL.
The idea is that the file itself serves as the lock. If it is present at startup time, then the second instance of the app should quit right away. So it must be a file that is created at startup, not one that is always present. That's also what creates a problem with the file approach - if the JVM crashes, the app or the JVM doesn't have a chance to delete the file, thus preventing the app from starting up after that.