When using the layout builder module, the contextual links module is a hard dependency. This works quite well in the context of the layout builder. But I don't want those links anywhere else, because elsewhere they are not helpful at best, and completly mess up my frontend theme at worst.

I've already written a custom theme negotiator to force my admin theme in layout building mode.

But I don't know how to remove all contextual links when I'm in frontend. My current ugly hack is

function mymodule_page_attachments(array &$page) {
  /**
   * Suppress the contextual edit pen on frontend themes, we only want it
   * for layout builder in admin theme
   */
  if (isset($build['#contextual_links'])) {
    $isAdminTheme = (\Drupal::service('theme.manager')->getActiveTheme()->getName() == \Drupal::config('system.theme')->get('admin'));

    if (!$isAdminTheme) {
      unset($build['#contextual_links']);
    }
    else {
      $routeName = \Drupal::routeMatch()->getRouteName();
      if (!in_array($routeName, ['layout_builder.overrides.node.view', 'layout_builder.defaults.node.view'])) {
        unset($build['#contextual_links']);
      }
    }
  }
}

and an additional rule in my frontend CSS

.contextual-toolbar-tab, [data-contextual-id] {
  display: none !important;
}

This does at least hide the contextual edit pen in frontend, but it does not really remove the contextual link menu. (There still is an empty placeholder div).

I've also tried @Shreya Shetty's suggestion

function mymodule_contextual_links_alter(array &$links, $group, array $route_parameters) {
  $routeName = \Drupal::routeMatch()->getRouteName();  //incorrect, this always returns "contextual.render"
  if (!in_array($routeName, ['layout_builder.overrides.node.view', 'layout_builder.defaults.node.view'])) {
    $links = [];
  }
}

In contextual alter I've got other problems: The current $routeName is always contextual.render and not the route of the host page, so I've no "entry point" to specifically alter non-layout bulder-pages. It also has the same problem as the hook_entity_view_alter workaround: The menu items are suppressed, but the empty contextual placeholder divs are still rendered.

Can anyone tell me how to reliably and cleanly remove the contextual links? I don't want them anywhere, except in layout building mode.