Don't put too much logic into your templates. Simply add the field template suggestion by implementing hook_theme_suggestions_HOOK_alter to respect view modes from within your theme or custom module:
/**
* Implements hook_theme_suggestions_HOOK_alter().
*/
function MYTHEME/MYMODULE_theme_suggestions_field_alter(array &$suggestions, array $variables) {
$element = $variables['element'];
$suggestions[] = 'field__' . $element['#entity_type'] . '__' . $element['#field_name'] . '__' . $element['#bundle'] . '__' . $element['#view_mode'];
}
Now you can add templates following this pattern: field--node--field-myfield--article--teaser.html.twig.
When using Layout Builder or Display Suite you might need to check what else you find inside $variables['element'] by printing its values with Kint (sub-module of Devel) via ksm($variables['element']) and then build the template suggestion from what you find there.