For entity reference fields there are 2 types of relationships added to views: a Referenced entity and a Referencing entity.
Referenced entity is used for a reference from the base table (e.g. A user view that references a fruit node).
Referencing entity is used for a reference to the base table (e.g. a fruit node view that references a user)
In your case, it's Referencing entity that you want.
Create a node view and a Entity Reference: Referencing entity relationship (the description should read A bridge to the User entity that is referencing Content via field_FRUIT_NODE_REF) that refers to your fruit node reference.
That should handle the data/reference part. What remains is aggregation/filtering (e.g. there will duplicates if fruit node references aren't distinct for each user) and display (e.g. how to display marking a fruit node referencing a current user).
Depending on your needs, you might need a custom field/template since there is a limit of what you can do with field basic rewriting and default user contextual filters.