Are you saying that you are using the not-yet-stable 7.x module Taxonomy Theme? Terms are assigned by Term Reference fields on fieldable entities. Since drupal is open source, there is always more than one way to do something. If you wanted to build a site that can do exactly as you have asked here, then I suppose it is possible to make a new entity that has a term reference and also embeds the view in its custom theme processes to get the fusion you are requesting. You will have to crawl into the Taxonomy Theme module code and see if it is using hook_node_view to do the check/change. If so, it would not fire on a straight view anyway but this custom entity (maintenance nightmare) should trigger it.
An easier method would be the template files that you dont want. Use theme_hook_suggestions where you have different tpl.php files for the different views based on path or whatever else you want. Different templates by path alias. You then just have the different templates use the different theme styles. I would also recommend putting the taxonomy term values as body level css classes so you can have one theme, and it just trigger different stylesheets based on term classes. Option1 | Option2 This would relieve your dependence on the dev-version module.
Option 3. In whatever theme is your default, copy the relative style sheets over from the other themes and just use the view-specific css class for each one, applying it manually. This means that you have duplicated style commands, so it will be a little harder to maintain what is coming from where, but at least you dont have to code any php.
Option 4. Sareed's comment about using block displays and embedding the views on other pages that have the terms. (...I should have thought of that)