Upon further investigation, it seems like [Peek Summary](http://drupal.org/project/peek_summary) might be exactly what you want. The screenshots on the documentation are for Drupal 6, and it's slightly different in Drupal 7, so I've attached some updated screenshots.

 - Choose Peek Summary from 'Contextual Filters' > 'When the filter value is not in the URL' > 'Display a summary' > 'Format'

![Choose Peek Summary from 'Contextual Filters' > 'When the filter value is not in the URL' > 'Display a summary' > 'Format'][1]

 - You'll see a view of sub-views. Click on the gear next to the pager for the group you'd like to limit.

![Click pager preferences icon][2]

 - Enter the max items to display on the pager options.  

![First field under Pager Options][3]

You'll have to edit the pager for each and every group you'd like to limit, which is unfortunate. It has the disadvantage of essentially creating [n] views for [n] groupings, which could be a performance issue depending on how many groupings you have, but it does limit the query to only the number set to display. This could improve your performance, depending on your use case. As always, cache responsibly.

  [1]: https://i.sstatic.net/HhlnD.jpg
  [2]: https://i.sstatic.net/TVa2w.jpg
  [3]: https://i.sstatic.net/SLgz5.jpg