I have a views index page that when displaying results without a text search I want to display sorted alphabetically by a taxonomy reference fields name. I can get it to sort by this reference field using the Search API Sort module, but it sorts by the term ID instead of based on the term name. I've also tried to do the same using Views sorting but it doesn't work. I tried creating a relationship to the term in Views but it wouldn't let me sort by fields added via that relationship.
Is this possible to do?