Is there a way to remove the dotted line with window operation system
I have no idea how to do this. The Windows LAF is contained in the WindowsSliderUI class. So you would have to look at that class to see if there is any way to customize the behaviour. However, this is not something I would recommend.
Again, this is the way the Windows LAF was designed to work.
In this case the dotted line is used to show the user which component has focus. This is the same for a JButton or JCheckBox. When a text component has focus you see a blinking caret.
Try other Windows applications and you will see this behaviour.
For example in IE you can go to Tools - Intenet Options - Security. Then tab to the "Security Level" slider. You will see the dotted line.
Windows users expect to see this dotted line.