The number of installed modules may influence performance. That's why, for example, Drupal core keeps in cache the name of the implemented hooks.
Given what Drupal core does, increasing the number of installed modules would have a lower impact on performance.
The aim for developers should not be reducing the number of modules/sub-modules, but writing modules that implement a specific feature. Clearly, if implementing a feature requires few lines, probably the module should be merged with another module, especially if that module is a sub-module.
To make an example, instead of the Node and User modules, there could be a single module (called, for example, Base Content). Since those modules are normally required from the installation profile, it would reduce the number of installed modules. It could also reduce the number of implemented hooks, since (for example)
node_user_predelete() could be merged with a class method or another function.
Anyway, doing so, two modules with different purposes would be merged together; the hooks could be reduced, but in some cases the code could now need to first check some conditions are met; probably more services should be lazy-loaded, which means that there would be two classes instead of one (the proxy class and the class implementing the service). Those classes would need to be synchronized with each other: Any change on the service class needs to be done on the proxy class too. (There is a script for doing that, but it works only on installed modules, which means the developer would need a local server running Drupal and remember to run the script for every lazy-loaded service.)
It's not just a matter of implemented hooks: In Drupal 9, a module that is complex enough implements many classes, some of which are service or plugin implementations. Even modules that are simple enough to just need a single route implement code in at least a class.