I'm currently working on my first Drupal 8 project and while many things are now integrated in the core, others still stump me (for example restoring options to now bare bones CKEditor requires like 4 additional modules). As I relied on Taxonomy Display module to display different taxonomies with different views and this module is not ported to D8 while similar module TVI is still in beta, I am left wondering what's the current standard of handling this use case. I could use template_preprocess_views_view to programatically switch displays depending on vid. Some suggested you can just use contextual filters to do the same, but I don't see how could you switch between different displays using contextual filters. To me it seems that multiple views would run at once, but some would return empty result sets, spamming database with duplicated queries. Is there really no cleaner way to do it?
You can use the option
Specify validation criteria of the Contextual filter to restrict your Views display to one or more Vocabularies, as you've heard before.
Then clone the display and do the same for different combinations of Vocabularies. You can change the path of each new page to avoid conflicts between different Views and calling multiple queries.
I don't know how Taxonomy display module works, I'm not sure if it improves performance, maybe you end up with the same performance metrics in your custom solution. The nodes still have to be matched with taxonomy in another database table, no matter which options you choose. I'm not a SQL expert, maybe someone else will be able to enlighten about the performance and how it affect Taxonomy term queries.