I'm doing the same thing for a media entity's route. My alterRoutes method is a little different - I'm using the [Route::setDefault()](https://api.drupal.org/api/drupal/vendor%21symfony%21routing%21Route.php/function/Route%3A%3AsetDefault/8.2.x) method:

    public function alterRoutes(RouteCollection $collection) {
      if ($route = $collection->get('entity.media.canonical')) {
        $route->setDefault('_controller', '\Drupal\my_module\Controller\MyController::view');
      }
    }

I think you may be unsetting the other defaults by passing the array like you are, but I'm not positive after looking at [Route::addDefaults()](https://api.drupal.org/api/drupal/vendor%21symfony%21routing%21Route.php/function/Route%3A%3AaddDefaults/8.2.x).