There might be a slight difference in performance but I guess it depends on your implementation. As you said yourself: it might be possible to replace a couple of db_select queries with EntityFieldQuery, although I can't really think of a situation where you'd be able to do that, if your db_selects are solid.
If I'm not mistaken EntityFieldQuery actually uses db_select or at least a SelectQuery to fetch data. I also think EntityFieldQuery looks up information about the entity that needs to be queried (ie: base table), this is cached though.
My guess is EntityFieldQuery would be somewhat slower, but the only way to see if you're losing any performance is by benchmarking (timing) it.
All in all I wouldn't worry about the performance of EntityFieldQuery. The biggest advantage of EntityFieldQuery is that it adds another layer of abstraction, making your code more loosely coupled and more cohesive. It's also pluggable so somebody could change the backend without you having to change any of your code.