I've included my original response below but it turns out I'm an idiot: All you have to do is create the taxonomy-term as a relationship and then you can just pick out the field you want. D'oh!
~~~ my original answer
Yeah, so this is a problem that is currently unsolvable without code as of Drupal 8.6.0. Unless you deploy Display Suite, which is using a sledgehammer to crack a nut.
There's a lot of discussion on drupal.org about the lack of control on title fields in respect to nodes but the issue goes deeper. It applies to Taxonomy as well: You cannot stop the title field from appearing in any view mode because that's the way the template works.
The only solution is to create a new template that doesn't display the title (in my case
taxonomy-term--book-series.twig.html) plus the supporting
hook_preprocess_taxonomy_term__book_series() all of which is a PITA.
Even then it's not great because there are no (default) theme suggestions for taxonomy terms that include view modes. It would have been much nicer if I could have designed a view mode called "description" and then had a theme suggestion of
taxonomy-term--description.twig.html because that would then work in all cases. Instead I have to create the same template and support code but for a different vocabulary.
All-in-all, not the best experience.