Make a new cell class that extends JPanel. Provide a setter and getter for the state (X, O, or blank). Override paintComponent to check the current state and paint an X, O, or just the background. This will be a very short and simple class. If you synchronize access to the setter, getter and paintComponent, you won't even need to worry about updating on the Event Dispatch
Thread. Every time you call the setter, also call repaint().
Create nine objects of this class to represent the nine tic tac toe cells. The containing panel can use a 3x3 GridLayout to hold these nine cells.