$render_controller->view() invokes [hook_ENTITY_TYPE_view()][1] and this would be a better place to add custom values to the entity view.

Although probably not good practice you could get your approach working by cloning the entity before modifying the field values:

    $entity = clone $entity;




  [1]: https://api.drupal.org/api/drupal/core%21lib%21Drupal%21Core%21Entity%21entity.api.php/function/hook_ENTITY_TYPE_view/8.7.x