It was a caching issue. This is my final code to cleanly hide all contextual links except in layout bulding mode:

use Drupal\Core\Entity\Display\EntityViewDisplayInterface;
use Drupal\Core\Entity\EntityInterface;

function mymodule_page_attachments(array &$page) {
  $routeName = \Drupal::routeMatch()->getRouteName();
  if (!in_array($routeName, ['layout_builder.overrides.node.view', 'layout_builder.defaults.node.view'])) {
    $page['#attached']['library'] = array_diff($page['#attached']['library'], ['contextual/drupal.contextual-links', 'contextual/drupal.contextual-toolbar']);
  }
}

function mymodule_entity_view_alter(array &$build, EntityInterface $entity, EntityViewDisplayInterface $display) {
  $routeName = \Drupal::routeMatch()->getRouteName();
  if (!in_array($routeName, ['layout_builder.overrides.node.view', 'layout_builder.defaults.node.view'])) {
    unset($build['#contextual_links']);
  }
}

For the caching issue:
The changes above most likely will (after cache clearing) remove all contextual link entries, but leave an empty placeholder like <div data-contextual-id="..." data-contextual-token="..."></div> in your rendered HTML. To get rid off those empty divs, go to your browsers Developer console > Application > Session storage and clear all keys starting with Drupal.contextual. After clearing them, the empty placeholder divs are gone too, no more CSS hacking needed.

It was a caching issue. This is my final code to cleanly hide all contextual links except in layout building mode:

use Drupal\Core\Entity\Display\EntityViewDisplayInterface;
use Drupal\Core\Entity\EntityInterface;

function mymodule_page_attachments(array &$page) {
  $routeName = \Drupal::routeMatch()->getRouteName();
  if (!in_array($routeName, ['layout_builder.overrides.node.view', 'layout_builder.defaults.node.view'])) {
    $page['#attached']['library'] = array_diff($page['#attached']['library'], ['contextual/drupal.contextual-links', 'contextual/drupal.contextual-toolbar']);
  }
}

function mymodule_entity_view_alter(array &$build, EntityInterface $entity, EntityViewDisplayInterface $display) {
  $routeName = \Drupal::routeMatch()->getRouteName();
  if (!in_array($routeName, ['layout_builder.overrides.node.view', 'layout_builder.defaults.node.view'])) {
    unset($build['#contextual_links']);
  }
}

For the caching issue:
The changes above most likely will (after cache clearing) remove all contextual link entries, but leave an empty placeholder like <div data-contextual-id="..." data-contextual-token="..."></div> in your rendered HTML. To get rid off those empty divs, go to your browsers Developer console > Application > Session storage and clear all keys starting with Drupal.contextual. After clearing them, the empty placeholder divs are gone too, no more CSS hacking needed.

It was a caching issue. This is the final code to cleanly hide all contextual links except in layout bulding mode:

use Drupal\Core\Entity\Display\EntityViewDisplayInterface;
use Drupal\Core\Entity\EntityInterface;

function mymodule_page_attachments(array &$page) {
  $routeName = \Drupal::routeMatch()->getRouteName();
  if (!in_array($routeName, ['layout_builder.overrides.node.view', 'layout_builder.defaults.node.view'])) {
    $page['#attached']['library'] = array_diff($page['#attached']['library'], ['contextual/drupal.contextual-links', 'contextual/drupal.contextual-toolbar']);
  }
}

function mymodule_entity_view_alter(array &$build, EntityInterface $entity, EntityViewDisplayInterface $display) {
  $routeName = \Drupal::routeMatch()->getRouteName();
  if (!in_array($routeName, ['layout_builder.overrides.node.view', 'layout_builder.defaults.node.view'])) {
    unset($build['#contextual_links']);
  }
}

For the caching issue:
The changes above most likely will (after cache clearing) remove all contextual link entries, but leave an empty placeholder like <div data-contextual-id="..." data-contextual-token="..."></div> in your rendered HTML. To get rid off those empty divs, go to your browsers Developer console > Application > Session storage and clear all keys starting with Drupal.contextual. After clearing them, the empty placeholder divs are gone too, no more CSS hacking needed.

It was a caching issue. This is my final code to cleanly hide all contextual links except in layout bulding mode:

use Drupal\Core\Entity\Display\EntityViewDisplayInterface;
use Drupal\Core\Entity\EntityInterface;

function mymodule_page_attachments(array &$page) {
  $routeName = \Drupal::routeMatch()->getRouteName();
  if (!in_array($routeName, ['layout_builder.overrides.node.view', 'layout_builder.defaults.node.view'])) {
    $page['#attached']['library'] = array_diff($page['#attached']['library'], ['contextual/drupal.contextual-links', 'contextual/drupal.contextual-toolbar']);
  }
}

function mymodule_entity_view_alter(array &$build, EntityInterface $entity, EntityViewDisplayInterface $display) {
  $routeName = \Drupal::routeMatch()->getRouteName();
  if (!in_array($routeName, ['layout_builder.overrides.node.view', 'layout_builder.defaults.node.view'])) {
    unset($build['#contextual_links']);
  }
}

For the caching issue:
The changes above most likely will (after cache clearing) remove all contextual link entries, but leave an empty placeholder like <div data-contextual-id="..." data-contextual-token="..."></div> in your rendered HTML. To get rid off those empty divs, go to your browsers Developer console > Application > Session storage and clear all keys starting with Drupal.contextual. After clearing them, the empty placeholder divs are gone too, no more CSS hacking needed.