I know that disabled modules do not effect performance. I also know that typically the number of modules you have enabled will effect the speed of you site. One individual on Drupal.org said you should never have more than 40 modules enabled. I found this strange because I would assume some modules are more intensive than others.
Is it simply the number of modules that somehow effect performance, (because the hook is always called), or is it really dependent on what you are doing in the module?