you may use hook_flag_access() here is my example...

function crystal_qa_flag_access($flag, $content_id, $action, $account) {
  //only the author of the question has right to use flag "Accepeted" on answers.
  //TODO: if i have no right use it,please hide it...
  if (($flag->name == 'accepted') && ($flag->content_type == 'node')  && ($node = node_load($content_id))) {
      if($node->type=='answer') { // the Answer node type machine name must be answer
          $question_nid = $node->field_answer['und']['0']['target_id'];
          $question = node_load($question_nid);
          return $question->uid==$account->uid ? TRUE : FALSE;
      }
  }
}

You may use hook_flag_access(), as in the following code.

 function crystal_qa_flag_access($flag, $entity_id, $action, $account) {
   // Only the author of the question has right to use flag "Accepeted" on answers.
   if (($flag->name == 'accepted') && ($flag->content_type == 'node')  && ($node = node_load($entity_id))) {
     if ($node->type=='answer') {
       // The content type machine name must be answer.
       $question_nid = $node->field_answer['und']['0']['target_id'];
       $question = node_load($question_nid);
       return ($question->uid == $account->uid);
     }
   }
 }