I think you may be unsetting some things in your alterRoutes method doing it the way you are. (However, I can't tell if or why that's happening after looking at Route::addDefaults().)
I'm doing the same thing for a media entity's route. My alterRoutes method is a little different - I'm using Route::setDefault().
public function alterRoutes(RouteCollection $collection) {
if ($route = $collection->get('entity.media.canonical')) {
$route->setDefault('_controller', '\Drupal\my_module\Controller\MyController::view');
}
}