$render_controller->view() invokes hook_ENTITY_TYPE_view() and this would be a better place to add custom values to the entity view.
Although probably not good practice you could get your approach working by cloning the entity before modifying the field values:
$entity = clone $entity;
In any case you need to adjust caching so that you store for each variant a different cache item in the render cache. For example by adding cache keys to the build array before returning it:
$build = $render_controller->view($entity, $view_mode);
if (isset($entity->foo)) {
$build['#cache']['keys'][] = $entity->foo;
}
return $build;