My understanding is that if I go the full panels route (i.e. every node is a panel) then I only need to worry about prepping views to populate the panels. Is this the right way to think about it? Or am I missing something in regards to where blocks are still relevant for site building under an all-panels site?
Well, it depends on what your needs are. Panels and Views is a very powerful combination that can handle almost anything. That said, there may still be times when you need or want a block. For example, the Heartbeat and Revisioning modules both supply useful blocks that I use on my (mostly Panels and Views) site. Yes, these two modules have Views integration, but their built-in blocks are already configured and easier to use than customizing views to meet my needs.
If you need a custom block and are trying to keep things in Panels, you can add a custom content pane. As a bonus, such panes can be exported with the Features module if you have a complicated deployment.
I'd say blocks can be very useful when combined with panels, for instance earlier today I had to place a graphic that linked to a newsletter sign up page in a right column of a page that is controlled by panels. I simply created a new block, uploaded an image using IMCE and CKEditor then pulled that block in using panels, much easier than creating a new region in a template.