The problem is that BoxLayout needs a reference to the container it's laying out. The container it lays out must be the same as the container that has this BoxLayout as its layout manager.
When you write frame.setLayout(new BoxLayout(frame, ...));BoxLayout has a reference to frame, but the frame doesn't have the box layout as it's layout manager, because frame just forwards the call to the content pane.
It was probably a mistake that JFrame forwards these calls. I almost always create my own JPanel and set a layout manager on it, and then set that panel as the content pane on a JFrame.