A factory just moves the logic you already have into another class with only that responsibility. In some applications that helps avoid duplicating the logic all over the place, but even if you do it just once it moves some details out of your main panel so it can focus on the things it really does well.
The map idea can help you change the factory logic from if-elseif-elseif to something simpler, like:
If you load the map from configuration you can add new panels without touching any of this code. "not changing code" is a very productive activity.