Can't understand Contextual Filter + Relationship Result in View

Question

Drupal 8

Goal When viewing a content type Story, I want a view to show other Story nodes that do not have the same taxonomy term as the Story that is being viewed. To achieve this I created a view with:

• Contextual Filter Content ID, Provide default value, Raw value from URL, Path component 2, More - Exclude so as to not have the viewed Story show up on the view
• Relationship from the Taxonomy Term ID referenced by a field in Story content type
• Contextual Filter Term ID that uses previous relationship: Provide default value, Taxonomy term ID from URL, Load default filter from term page, Load default filter from node page, Limit terms by vocabulary, More - Exclude

And it works, sort of. When accessing a Story that has at least one taxonomy term associated to it, the view shows other stories that don't have the same taxonomy term. But, when viewing a story that has no taxonomy term associated to it, the view shows nothing.

Am I wrong to think that the view should be showing all the results?