The more modules you add to a site, the more code that likely has to execute, which equates to more overhead per request. Organic Groups particularly can be heavy, depending on what you are trying to achieve. I do not know how much progress they've made in the last few years, but in my experience it was always pretty heavy.
Logged in users also bypass caching, whereas anonymous users see as much cache as possible. This isn't as true for 8.x, but the differences in 7.x can be pretty black and white. Juggling multiple logged in users simultaneously can put a hurt on low end setups or especially if you haven't implemented caching. Or a runaway module could be calling functions every single request per user, adding to that time.
Honestly, the best tools that will help you pin down potential bottlenecks are xhprof and using a service like New Relic. It will give you far more intel about your application than anyone here ever could.
Other things you can do simply comes down to good planning. Are pages made up of lots of Views? Implement Views caching on displays that don't change often. Are you using Panels? Do you need to use Panels? Are you running memcache, does PHP have 256M memory limit, is your server underpowered?
I would start with New Relic, honestly, and xhprof. New Relic is easier to setup than xhprof, depending on your experience. You will need to profile your site before you know what steps to take first.
If you are running Drupal 8, also install the Web Profiler toolbar included with Devel.
