Panels adds additional classes and divs which are unnecessary for my purpose. How can I remove these?
There are a couple ways you can tackle this.
One option is to simply create your own panel layouts. I often do this because I like control over the markup and I want to reduce the divitus in my theme. It also lets me create layouts that more directly match what I need. It's pretty easy to create your own panel layouts in your theme. Read more here.
Another option is to override the panel template files in your theme. Similar to any module that provides a template file, you can copy that template file to your theme and change the markup. You'll find the panel template files in [path_to_modules]/panels/plugins/layouts/[each_layout].
Why don't you use Clean Markup Module.
This module aims to clean up, enhance and facilitate the customization of markup for Drupal core and several popular contrib modules such as Panels.