Caveat : This is my experience as an advanced Site Builder, not developer.
You do not have to use Product Displays if you are using Panels. Using Panels instead of Product Displays is often done to replace Product Displays if you only have one display per product, so you don not need to replicate your Product -> Product Display.
In your Panel setup, you need to create a page that uses the Product ID as a variable in your URL. Create a URL with a variable (e.g. /catalog/%products), then in the Arguments section you configure %products to use Commerce Product ID.
Then create your Page with the Panel layout desired, inserting fields where needed. In order to get a "Add to Cart" button, what I've done is create a single-item View of Product fields, and display the Add to Cart button in the Field list. I add a Contextual Filter (in Advanced) to fetch the correct Product ID value (Provide Default Value -> Raw value from URL). I know there is another way to do this with a Views moudle for Panels, but I don't know the specifics - maybe someone else can help?
Of course there are more complicated things you may need to solve - multiple images, related products - but these can be solved with standard Drupal solutions (Views, Cloud Zoom / Gallery, etc).