Image inside paragraph being cached

Question

I have a Paragraph type called image - that holds a media reference and also a field which describes how the sibling image should be rendered. I look at the value of the "field_image_style" - and then apply a responsive image style to the Image in a preprocess_paragraph call:

function my_module_preprocess_paragraph(&$variables)
{

  if ($variables['paragraph']->hasField('field_image_style') && isset($variables['paragraph']->field_image_style->entity)) {
    /** @var \Drupal\Core\Entity\Entity\EntityViewMode $viewMode */
    $viewMode = $variables['paragraph']->field_image_style->entity->field_view_mode_for_media->entity;
    //kint($viewMode);
    if ($viewMode != null) {
      $viewModeMachineName = $viewMode->id();
      //kint($viewModeMachineName);
      $viewModeMachineNameMediaPrefixRemove = str_replace("media.", "", $viewModeMachineName);
      //kint($viewModeMachineNameMediaPrefixRemove);

      //kint($variables['content']['field_image'][0]['#view_mode']);
      $variables['content']['field_image'][0]['#view_mode'] = $viewModeMachineNameMediaPrefixRemove;
    }
  }
}

The html for rendering the image is being cached - and when I rebuild the cache - the site flips between one flavour and the next. How do I make the cache key on the value I am setting such that I get a cached copy for each different $viewModeMachineNameMediaPrefixRemove ?

Full data structure added below for completeness:

Paragraph type: Image Image contains two fields - both Entity references. field_image > Entity Reference > Media field_image_style > Entity Reference > Taxonomy Term (Image style)

The Taxonomy term Image style then has two fields; Tax term > Text plain (css format) - used to decorate the paragraph Tax term > Entity Reference > View mode (points to view mode of Media Image)

This view mode is then used to select primarily one of two Responsive Image Styles.

The preprocess hook is on the paragraph - as the paragraph knows about both the image and the way in which it needs to be rendered.

Answer 1

Thanks to all for their contributions. Ended up using the solution proposed by @Hudri:

unset($variables['content']['field_image'][0]['#cache']['keys']);

Which allowed for both image styles to be used on the same image across different paragraphs.