I have a entity reference field formatter that provides #cache tags even for entities that are not displayed since I want to flush the render cache of the entity that has this field attached when the referenced entity changes so it will be hidden/displayed immediately(note that I am not using the checkAccess() for this).
The issue I have is that if I provide empty entry for such entity only with the cache tags set, it will result in empty field item div to be rendered. If I put the cache tags to the parent element(the element that is returned by the viewElements() method) and not "delta" item within this element, it won't work at all. I'm not sure if this is because the cache is being overwritten later on or if that is because of the way Drupal core works or whatnot but it is just not working.
So my question is: how could I provide these cache tags and avoid having empty field item divs in the resulting html output?