This is not a straight-forward flag trim. Trim works on all content in the set of a content type, and I don't see how you can do it on a subset.
This answer uses Views bulk Operations, so please install that. I also assume the flag is global (or if not then only one user flags)
We will create a view that will get the list of images to unflag. Then we will create a rule component that will unflag the images, and lastly we will create a rule that will initiate all of this when an image is flagged.
- Create a view that will fetch the images that belong to a certain gallery and that have been flagged.
To do that, create a view (master display only) on content (node), and filter on content type (image). Add three relationships:
Relationship 1: Add a relationship 'Flags: Node flag', include only flagged content, set the flag to the desired flag, and set it to 'Any User'. This will ensure that only flagged images show up.
Relationship 2: 'A bridge to the Content entity that is referenced via field_gallery'. This will bring in the referenced 'gallery'.
Relationship 3: This one may seem weird, but bear with me. Add 'A bridge to the Content entity that is referencing Content via field_gallery', and set its relationship to the previous one, ie 'Content entity referenced from field_gallery'. This links back to the image type. The reason for this will become apparent in the next step.
Add a contextual filter of 'Content nid' and set its relationship to the third relationship you added above, ie 'Content referencing Content from field_gallery'. We will pass in an image nid, get the associated gallery, and then get all the associated images belonging to that gallery that are flagged. So now we have our list that we will use in rules.
The last step is to add a new field (the only field we need). Add a field 'Bulk operations: Content'. Do not use a relationship. This will allow us to use the view with rules.
Finally, remove the pager as it is not needed.
Now we need to add a rule component and a rule.
- Create a rule component.
Add a new rule component called 'Unflag Image'. It is of plugin type 'Rule'. Add two variables.
First variable: Data type -> node, Label -> previously flagged image, Machine name -> prev_image
Second variable: Data type -> node, Label -> newly flagged image, Machine name -> new_image
Now add a condition: Data comparison. Compare 'prev-image:nid' equal 'new-image:nid' and tick 'Negate'. This will check that the previously flagged image does not equal the newly flagged image.
Add an action: Unlag a node, and set the node to flag as 'prev-image'. Set user on whose behalf to flag as 'site:current-user', and select skip permission check: false.
- Create a Rule.
Add a rule that has event: 'A node has been flagged' and select your flag.
There is no condition.
Add an action to the rule: Load a list of entity objects from a VBO View. For its value select the view that you added previously. For the argument select the flagged-node:nid. Leave the provide variables as is.
Add a loop, and select 'entity list' (the provided variable from previous step), and leave defaults for current list item.
Inside the loop, add an action: Component: Rule: Unflag image. This is the component we added earlier. For previously added image, select 'list-item' (provided variable of previous step), and for newly flagged image, select 'flagged node'.
And that is it. You should now debug it as I have not checked it, but the logic is more or less there and you can take it further.