What is the basic concept behind hooks? refers to modulename.api.php.
Does this apply to Drupal 8? What is this file? Is there an example of how to code it somewhere?
.api.php file contains the hooks that a certain module supports, and is available for reference.
It offers documentation and example code on how to use the hooks in other modules. It is not specific to Drupal 8 as several modules in Drupal 7 offer it as well.
It does not actually implement or call the hooks and is for reference only.
As for an example of how to write it, it depends on the different hooks your module implements. Explain the different input and return parameters, adding additional data alterations as you see fit.
search.api.php is one of many examples in core. Hooks are either invoked, as for example, in common.inc:
$updaters = \Drupal::moduleHandler()->invokeAll('updater_info');
That is the the current way in procedural code. In object-oriented terms, with Dependency Injection, refer to this example in ModuleInstaller:
There are also alter hooks, like this one in BreadCrumbManager:
$this->moduleHandler->alter('system_breadcrumb', $breadcrumb, $route_match, $context);