I am afraid, it's impossible to achieve what you want with a simple entity reference field alone.
The problem is that while the wardobe content saves the date of its most recent edit, you can not tell which (if any) product was added during that editing session (unless you use versioning consistently and compare the products saved in different versions manually). To complicate things further, an entity only saves the date of its most recent edit (saving over the previous edit date) which makes it very inadequate to track when a particular product was added to a wardrobe.
The biggest issue you face is that the information you need (when an item is added) is not directly tracked at all. While you could probably figure it out if you use versioning, this data would still be very hard to access in views.
What you can do is make a list of wardobes a particular item is part of and order them by most recently updated wardobe (though this most recent edit may or may not have actually added that product). If that's not what you want you need to find a way to track the missing date information yourself.
Possible starting points which come to mind:
- Custom compound field with joined reference + date fields: https://www.phase2technology.com/blog/compound-fields-in-drupal-7/
- Custom module which adds a db table with columns for wardrobe id, product id and date. You can access these data on node save, during which you can still access the old version of the wardobe, check which (if any) products were added, and add those entires to the db.