Code-wise, both Views and displays are represented by objects. When a View is being processed, it loads all it's child displays into memory, regardless if you are rendering only one View in the page you are looking at. While database queries are executed only for the active display, the properties of the inactive displays are still loaded into memory.
This means that if you have 50 displays in one View, 50 displays will be loaded into memory, regardless if you are only displaying only one of them. So the answer to your question is yes, typically the more displays in a view you have, the more inefficient the View becomes memory-wise.
With that being said, I personally think that having anywhere between 2-10 displays in a View should prove negligible memory-wise. Having a discreet amount of displays in a View does serve it's practical purposes, after all (otherwise, why would they be there in the first place?).
As to your second question (performance impact of additional filters, etc), yes, adding certain parameters to your View will affect performance. If you want to know by exactly how much, you can go to admin/structure/views/settings (if you are in D7) and select Show information and statistics about the view during live preview, Show the SQL query, and Show performance statistics. Here is a screenshot of the performance screen:
If you are further interested in the behavior of the queries (how many rows are analyzed, which keys are used), you can copy the query from the Views interface (once you've selected "Show SQL query"), and analyze the query using the MySQL
explain command. There's plenty of articles like this one for
explain usage on Google.