I've run into issues like this in the past. I'm not 100% sure this is correct but you may try this and see if it works for you.
One of the issues of aggregating by dates is that the Views aggregation happens during the database query stage, but the display of "Time ago" happens later, during the rendering stage. Therefore, when the date is converted to "years ago" it's already too late for aggregation.
I've found that Views Date Format SQL has helped me solve this issue in the past. This module...
The Views Date Format SQL module allows to format date fields using SQL. This enables group aggregation for date fields using the choosen granularity.
The core functionality is to remove the date formatting from render()
and put it in query(). I.e. format date values using SQL's DATE_FORMAT
rather than PHP's format_date. This is achieved by assigning a new
default handler to node 'created' and 'changed' date fields. This
handler extends and overrides views's build in
views_handler_field_date.
Another one to try if that doesn't work is to avoid using Views aggregate
option altogether and use a post-query aggregation like Views Aggregator Plus.