In Drupal 8, what is the correct way of returning access denied to a user (outside of access requirement on a controller)?
I'd say it is to throw an AccessDeniedHttpException most of the time, but I have seen modules set an event response to 403, or do a redirect response to system.403.
Generally, the correct way is to throw new AccessDeniedHttpException() over setting a 403 or a redirect response, which bypasses the exception handling, so that it's no longer possible to configure a different handling of 403's in contrib or custom code. Unless you want to bypass this intentionally, for example if you want to produce a faster 403.
However, in most cases it is preferable to control the access as route requirement. The main effect is the same, the routing system throws an exception as well. But it does other things, like controlling the visibility of menu links, which you don't get when you throw the exception in custom code.
system.403, without a redirect, but with a subrequest keeping the original URL in the address bar.