Okay, well the major issue with this particular field is that id doesn't have all the properties you need to go through the standard routes to display the term instead of the id.
So, without a major rewrite of the module I would do this if you really need this feature.
Copy the views-view-summary-unformatted.tpl.php to your admin theme directory and edit it to include a large switch statement with all of the country codes you are interested in, probably copied in from the module itself. Here is a working example of one such modification for your example of Japan. Obviously this is not ideal, but without altering the field information of the module there is no good way to link the Key used here with the value. Sorry there does not seem to be a better way.
<?php
/**
* @file views-view-summary-unformatted.tpl.php
*/
?>
<?php foreach ($rows as $id => $row): ?>
<?php print (!empty($options['inline']) ? '<span' : '<div') . ' class="views-summary views-summary-unformatted">'; ?>
<?php if (!empty($row->separator)) { print $row->separator; } ?>
<a href="<?php print $row->url; ?>"<?php print !empty($row_classes[$id]) ? ' class="' . $row_classes[$id] . '"' : ''; ?>>
<?php
include_once DRUPAL_ROOT . '/includes/locale.inc';
$countries = country_get_list();
if($countries[$row->link]) {
print $countries[$row->link];
}
else {
print $row->link;
}
?></a>
<?php if (!empty($options['count'])): ?>
(<?php print $row->count; ?>)
<?php endif; ?>
<?php print !empty($options['inline']) ? '</span>' : '</div>'; ?>
<?php endforeach; ?>