A way to limit this in code would be to add something like the following to a custom module:
function custom_views_pre_render(&$view) {
//get the rows from the view just before render
$results = $view->result;
//create a counter
$count = '';
//we're going to built up a new $result array
$new_results = array();
//iterate through each view row
foreach($results as $result) {
//find the taxonomy term
$term = $result->taxonomy_term_data_name;
//add the term to a string of all the terms we've seen so far
$count .= $term;
//make sure to separate them with spaces to make them easier to count
$count .= ' ' ;
//count how many rows have the same term as the current one
$term_count = array_count_values(str_word_count($count, 1));
if($term_count[$term] <= 3){
//if this is the third or fewer row with this term, add it to the new result array
$new_results[] = $result;
}
}
//instead of the normal view output, only show the results we put in our array.
$view->result = $new_results;
}
This is for a view of taxonomy terms that is connected to nodes via a relationship. If you just have a view of nodes, your mileage may vary.
Though it prevents the display of more than 3 per term, this will not prevent the query from returning all the results for each term, so it doesn't improve SQL performance at all. If you have a very large number of results for each term, making separate view panel displays and putting them all in one region using something like CTools Page Manager so you're not running huge queries.
As always, you're going to want to cache this stuff on production.