Sounds like something like this should work for you, to set the file display for the image file type, view mode name VIEW_MODE (replace with your custom view mode), to display original image:

$display = array(
    'api_version' => 1,
     'name' => 'image__VIEW_MODE__file_field_image',
    'status' => 1,
    'settings' => array(
        'image_style' => '',
    ),
);
file_display_save((object) $display);

Sounds like something like this should work for you, to set the file display for the image file type, view mode name VIEW_MODE (replace with your custom view mode), to display original image. It either adds a new file display or updates an existing one:

$displays = file_displays_load('image', 'VIEW_MODE', TRUE);
$formatter = 'file_field_image';
if (array_key_exists($formatter, $displays)) {
  $display = $displays[$formatter];
}
else {
  $display = (object) array(
    'api_version' => 1,
    'name' => 'image__VIEW_MODE__' . $formatter,
  );
}
$display->status = 1;
$display->settings['image_style'] = '';
file_display_save($display);