Benefit of config entities over regular config entries?

Question

In Drupal 8 it is possible to store configuration in "config entities".

However, the same can also be achieved with non-entity config entries.

E.g. one example for a config entity schema, from core.entity.schema.yml:

core.entity_view_display.*.*.*:
  type: config_entity
  label: 'Entity display'
  mapping:
    id:
      type: string
      label: 'ID'
    targetEntityType:
      type: string
      label: 'Target entity type'
    bundle:
      type: string
      label: 'Bundle'
    mode:
      type: string
      label: 'View or form mode machine name'
    content:
      type: sequence
      ...
    hidden:
      type: sequence
      ...

The same information could be stored in entries with type: mapping, like so:

core.entity_view_display.*.*.*:
  type: mapping
  label: 'Entity display'
  mapping:
    content:
      type: sequence
      ...
    hidden:
      type: sequence
      ...

In fact, we would not even have to store the 'targetEntityType', 'bundle' and 'mode' as properties, because these are already part of the key (the asterisks in the schema key). We could also omit the 'id', because the key is already a unique identifier, making the id redundant.

Obviously, one would not use entity crud components to deal with those config entries, but call directly the config API, e.g. \Drupal::configFactory->get(..).

Also, one would not inherit from EntityForm, but have custom form classes. But it seems all doable.

Question

So: What would we miss by not using config entities? In which cases would a simple "mapping" be sufficient?

To stick with the example: What can we currently do with 'entity_view_display' entities, that would not be possible if these were just 'mapping' entries?

Background / Use case

My original use case was this one: Custom entries in entity view modes?, which came up when I wanted to port entdispfieldui to Drupal 8.

This question was very specific, so I wanted to ask a more general question about the benefits of config entities.

In this case I decided to use non-entity config entries. One reason was that config entities caused recursion in hook_entity_extra_field_info().

I am going to port more modules to Drupal 8, where the same question will come up, but where I don't have to worry about hook_entity_extra_field_info(). E.g. cfrpreset.