In the
Java in General (intermediate) forum there is at least one good topic about only allowing one instance of an application. If you use the search I'm sure you'll be able to find it.
Inside your application it's a lot easier. Keep a mapping of users to their chat windows. If you want to open a new chat window, check if there already is a chat window for the user. Only if it's not the case
you should create the window and of course add the mapping.
Don't forget to remove the mapping if the window is closed though (tip: use a WindowListener).