You should be able to do this using the Layered Pane, but what happens when the dialog is visible outside the bounds of the frame?
If you want to make your application unusable, it's probably best to indicate this in the way the GUI looks as well, otherwise it will seem the application has crashed. Simply disable the frame and the dialogs using setEnabled(false).
Another way you could do this is create a modal dialog, which says that the program has locked. If you close the modal dialog, you could make the entire application exit.
What is it exactly that you want to achieve, and why? There are several ways to do it, but some are more suitable than others.