GridLayout totally ignores the component's size.
From the API docs for GridLayout
The container is divided into equal-sized rectangles, and one component is placed in each rectangle
To fix your problem,
1) Change from GridLayout to FlowLayout
2) For the panels, use setPreferredSize() instead of setSize().
PS. In your code you haven't set the preferred size for the first panel.