You should profile this and see. It totally depends on your system.
Install the Devel and Performance modules. Then enable the Query log (in the D7 version it is at "admin/config/development/devel"). Visit some view pages, and look at the queries at the bottom of the page. Tweak your views and re-run. Just be sure to disable caching (including MySQL query caching) when you do this, as it may skew results.
You can also copy/paste the SQL from the view config page (enable from "admin/structure/views/settings"), into Query Browser or another tool and look at the execution time. If you do this, bypass the query cache, to make sure you get accurate results.
Also keep in mind that buffer sizes, etc, can affect results since data may or may not need to be read from disk. You should really have an optimized MySQL instance before you profile.