I think menu_get_object() might be helpful:
menu_get_object() provides access to objects loaded by the current router item. For example, on the page node/%node, the router loads the %node object, and calling menu_get_object() will return that. Normally, it is necessary to specify the type of object referenced, however node is the default. The following example tests to see whether the node being displayed is of the "story" content type:
<?php
$node = menu_get_object();
$story = $node->type == 'story';
?>
You can use this with any router-loaded object so it stands to reason that on entity pages (including taxonomy term pages) that this would get you the correct object:
$entity = menu_get_object($entity_type);
UPDATE
On further inspection you still have to pass the position of the ID in the URL as a parameter to menu_get_object() so it's pretty much the same as using arg(), but it's arguably a bit cleaner. For taxonomy terms the path is taxonomy/term/% so you would call:
$term = menu_get_object('taxonomy_term', 2);
It's worth bearing in mind that entity_load will only incur a database call if the entity hasn't previously been loaded in the current page request. If it's already been loaded, it will be served from the static cache.
UPDATE 2
Just to add an example of the static pattern to avoid reloading the entity over and over again if you're getting memory errors:
$loaded_entity = &drupal_static(__FUNCTION__, NULL);
if (!$loaded_entity) {
$loaded_entity = menu_get_object();
}
return $loaded_entity;