A FlowLayout is not a good layout to use for this. It attempts to display components on a single line and will potentially wrap when the width is too small. You can't control this wrapping as the user resizes the frame. Try adding your panel to the CENTER of the Borderlayout and you will see what I mean.
You need to use a more sophisticated layout manager, or nest multiple panels with different layout managers. Check out the Swing tutorial on Using Layout Managers for more information and examples.
For something quick and dirty you can use a GridLayout
, but this will make all components the same size. A GridBagLayout
is more complicated but you can control the size of individual components better.