It's not possible to enable the layout builder without an entity which is supporting view modes.

However, it seems like you have such an entity already. So you could add a view mode at *admin/structure/display-modes/view/add/taxonomy_term*. Then visit *admin/structure/taxonomy*, select **Manage Display** of a vocabulary and enable the view mode in **Custom display settings** at the bottom of the page. Save and select the tab of the new view mode at the top. Check *Use Layout Builder* in **Layout options**, save again and click the button "Manage Layout".

In the controller render the term with the new view mode:

      public function taxonomyPage($term) {
        // load the term or enhance the route to load the term automatically
        // See www.drupal.org/docs/8/api/routing-system/parameters-in-routes/using-parameters-in-routes
        return $this->entityTypeManager()
          ->getViewBuilder('taxonomy_term')
          ->view($term, 'my_view_mode');
      }