I tried to define a custom field formatter but it didn't find the hook.
In the end it was much easier to use the Entity Reference Views Formatter module, then theme the view field to add the desired behavior to the relevant link. I used the rules_link module to define the untag link, then added the untag link into the field's theme. Next time I should probably use a tpl.php file as well.
File: views-view-field--with--block--field-with.tpl.php
<?php
global $user;
$my_uid = $user->uid;
$node = node_load ($row->nid);
$results = db_query( // Load the field without loading the entire node.
"SELECT field_with_target_id
FROM field_data_field_with
WHERE entity_id = :entity_id",
array(':entity_id' =>$row->nid));
$lines = "";
foreach($results as $result) {
$uid = $result->field_with_target_id;
$account = user_load((int)$uid);
$rendered = l(mymodule_get_realname($account), "user/$uid");
$untag = "";
if ($uid == $my_uid) {
$rules_link = rules_link_load('un_tag_me');
$untag = render(rules_link_render_link($rules_link,$row->nid));
$untag = str_replace (
array("<div ", 'title="x"'),
array("<div title='Un-tag me'", 'title="Un-tag me"'),
$untag);
}
$line = "<div class='field-item'>$rendered $untag</div>";
$lines .= $line;
}
if (!empty($lines)) {
$output = "<div class='field-items'>$lines</div>";
}
print $output;
?>