When you display a certain number of records per page it does not load and hide the extra records, it simply never pulls them from the database. So you'd certainly be helping performance by limiting the number that are displayed per page.
In fact, if you look at the SQL query generated by the view, it'll say the following if you're showing 20 items per page, for example:
LIMIT 20 OFFSET 0
====================== EDIT ====================
What this means for performance
As @Clive and I were discussing above in the comments, A LIMITed query will inevitably be faster than one with no LIMIT, but overall performance will also inevitably decrease as the number of nodes you are searching through increases.
Limiting the number of returned records has performance savings on the Drupal side too, though. That's because Drupal wouldn't be running the additional processing on the Drupal side to do things like access checks, pre-processing and other display-related functions for the records that aren't returned by the query.
p.s. You can configure whether you want Views to show you the sql query it's generating by going to /admin/structure/views/settings and checking the 'Show the SQL query' checkbox in the 'Live Preview Settings' section.