Upon further investigation, it seems like Peek Summary might be exactly what you want. The screenshots on the documentation are for Drupal 6, and it's slightly different in Drupal 7, so I've attached some updated screenshots.
- Choose Peek Summary from 'Contextual Filters' > 'When the filter value is not in the URL' > 'Display a summary' > 'Format'
- You'll see a view of sub-views. Click on the gear next to the pager for the group you'd like to limit.
- Enter the max items to display on the pager options.
You'll have to edit the pager for each and every group you'd like to limit, which is unfortunate. When you edit the pager for one group, it changes the settings for all groups. If you want to display a number of nodes per group different than the total number of groups, override that in the 'Format' settings under 'Contextual Filters' (on first screenshot in purple).
It has the disadvantage of essentially creating [n] views for [n] groupings, which could be a performance issue depending on how many groupings you have, but it does limit the query to only the number set to display. This could improve your performance, depending on your use case. As always, cache responsibly.