How to make a block vocabulary-specific
You can use the Block Visibility Groups module in conjunction with the Vocabulary Condition module (linked on that page). Using those two modules together will give you a checkbox on the block to pick the vocabulary it should be shown for.
If you want to do it programmatically, the Vocabulary Condition module should be a good code reference. There's also another Vocabulary Condition module (same name, ugh!) which could also be a good reference.
Alternative idea: Use different view pages for each vocabulary
If you're using the default taxonomy term view provided by core, I'm pretty sure you can customize it so that you have one page for terms with the Product vocabulary and another for the Blog category. Then you can simply expose the filter on the view page for the relevant vocabulary.