Yes, the multisite system can do that out of the box. From the Site-specific modules and themes section of the multisite docs:
Each site configuration can have its own site-specific modules and themes in addition to those installed in the standard 'modules' and 'themes' directories. To use site-specific modules or themes, simply create a 'modules' or 'themes' directory within the site configuration directory. For example, if sub.example.com has a custom theme and a custom module that should not be accessible to other sites, the setup would look like this:
So just make sure you don't put modules that shouldn't be shared in /sites/all/modules; put them in /sites/SITE/modules instead.
Can site A can "block" (for lack of a better term) a module installed by site B, in which site B installs to sites/all/modules?
You could implement
hook_system_info_alter in site-exclusive modules to prevent certain ones from ever being displayed to the user on those sites, but as there's no record anywhere of what site downloaded/installed a module to /sites/all/modules, you'd have to keep those modules up to date manually. Unless you supplement that with an API exposed from each site that let each other site know what it had installed and what should/shouldn't be available to install.
Is best practice to configure specific modules per site rather than trying to prevent the transposition?
I think it's fair to say an inclusive rather than exclusive approach is better practice here, any other approach is a bit awkward. It gets more complicated the more automated/robust you want an exclusive approach to be, whereas doing it the other way round just works.