I don't have enough points to leave a simple comment. I haven't tested this yet. However, I think that you need to select in the 'Relationships' section the node reference. This will join the two tables and give the view access to all the fields in both. Then add an argument and when configuring the argument use the relationship you choose. I'm not sure which argument is best to choose but I think that node title is a good one for the summary and then select the Summery, ascending or desc. This actually creates a list of all the possible titles of A with the number of node B attached to each with a number next to is. When the title of node A is clicked in the block or page view it opens the page with node A's title as the big page title and all of fields of B as long as you selected the relationship under each fields setting.
I wish I had time right now to test it. It's only a theory but I'm sure it's the general direction of solving this.