Referenced entity is output each time instead of once

Question

I use out-of-the-box articles content type to which a I added en entity reference field so I can assign multiple users to that article. I used this reference in order to be able to display multiple authors for that article.

The end goal is, that when the detailpage of an article is displayed, to have a block listing the assigned users and per user listing the nodetitles that reference this users (so-called authors). Basically, trying to mimic the out-of-the-box author-functionality using entityreference. In views' preview function, I would then add a nid that uses the conextual fileter.

The output that I want then would look as follows:

From John Doe:
Lorem Ipsum
Ipsum Lorem
Irem Lopsum

From Plain Jane:
Irem Lopsum
Ipsum Lorem

I almost managed to configure this with views

1. Added a contextual filter that pulls in the nid so I can filter the assigned users based on that nid.
2. Configured a relationship for the users that are referenced by the nid that is displayed. I use this in my first field to display the assigned users. This works fine
3. Configured a second relationship for the nodes referencing my users. Also pulled in this relationship when making my second field that is content:title.

What I get instead though and what I don't want is the following:

From John Doe:
Lorem Ipsum
From John Doe:
Ipsum Lorem
From John Doe:
Irem Lopsum

From Plain Jane:
Irem Lopsum
From Plain Jane:
Ipsum Lorem

I read this drupal answer, where someone wants to display an amount of referenced entities. This seems close to my issue, but I have the feeling that my problem isn't really one with aggregation or grouping. Still not sure of course, but I just can't seem to understand how it would fit my case. Semantically though, the word grouping obviously suggests I'm on the right track. Technically OTOH, it's not quite making sense to me, since it seems to group data rather than markup the way I want it to.

I also watched both the NodeOne video's 1 and 2 mentioned in drupal answer stated above. Other than fiddling with aggregation settings, I also deleted the default sort criteria as suggested. Didn't help.