It's not possible to enable the layout builder without an entity which is supporting view modes.
However, it seems like you have such an entity already. So you could add a view mode at admin/structure/display-modes/view/add/taxonomy_term. Then visit admin/structure/taxonomy, select Manage Display of a vocabulary and enable the view mode in Custom display settings at the bottom of the page. Save and after the form reloads select the tab of the new view mode at the top. Check Use Layout Builder in Layout options, save again and click the button "Manage Layout".
In the controller render the term in the new view mode:
$build = \Drupal::entityTypeManager()->getViewBuilder('taxonomy_term')->view($term, $view_mode);