Ughh.. apparently my question had the answer all along. When I add the code snippet:

  $languageManager = \Drupal::languageManager();
  $langcode = $languageManager->getCurrentLanguage(LanguageInterface::TYPE_CONTENT)->getId();
  $language = $languageManager->getLanguage($langcode);
  $languageManager->setConfigOverrideLanguage($language);

to a _views_pre_render hook it does what I want and uses the config override language to override the language used for the configurable parts of the view (title, column headings and more). The only thing it doesn't handle is the dates - as these are just done wrong in the Datetime module. Not sure why I didn't see this working when I tried it earlier.

Ughh.. apparently my question had the answer all along. When I add the code snippet:

  $languageManager = \Drupal::languageManager();
  $langcode = $languageManager->getCurrentLanguage(LanguageInterface::TYPE_CONTENT)->getId();
  $language = $languageManager->getLanguage($langcode);
  $languageManager->setConfigOverrideLanguage($language);

to a _views_pre_render hook it does what I want and uses the config override language to override the language used for the configurable parts of the view (title, column headings and more). The only thing it doesn't handle is the dates - as these are just done wrong in the Datetime module. Not sure why I didn't see this working when I tried it earlier.

And I can override the DateFormatter core service to use the Content language instead of the default (which is interface) and this fixes dates everywhere on the site. This should likely be submitted as a patch to core as I am sure this is a bug - why would dates be considered interface differently than all other content?