Where is it documented that fields included in the entity_keys are 'required' regardless of the required property value in the field definition?

Question

I've created a custom entity. One of the fields I've added is an entity reference field that was created calling BaseFieldDefinition::setRequired(FALSE). When I installed my module and tried to add a new instance of my custom entity, I got an error when that field was not set, even though it was defined as not required.

For reference, this is the field definition I'm using.

$fields['publisher'] = BaseFieldDefinition::create('entity_reference')
  ->setLabel(t('Publisher'))
  ->setRequired(FALSE)
  ->setSetting('target_type', 'taxonomy_term')
  ->setSetting('handler', 'default:taxonomy_term')
  ->setSetting('handler_settings', array(
    'target_bundles' => array(
      'publisher' => 'publisher'
    ),
    // 'sort' => array(
    //  'field' => 'name'
    // )
  ))
  ->setDisplayOptions('view', array(
    'label' => 'hidden',
    'type' => 'author',
    'weight' => 0,
  ))
  ->setDisplayOptions('form', array(
    'title' => t('Pubs'),
    'type' => 'options_select',
    'weight' => 4,
  ))
  ->setDisplayConfigurable('form', TRUE)
  ->setDisplayConfigurable('view', TRUE);

This is the error I got.

Drupal\Core\Entity\EntityStorageException: SQLSTATE[23000]: Integrity constraint violation: 1048 Column 'publisher' cannot be null: INSERT INTO "series"...

The database table for this custom entity contains the following fields.

It's evident that, even though this field has been created calling setRequired(FALSE), it must be set.

To solve this issue, I found that I can manually update my database table's properties so that the field's 'Null' property is set to 'Yes', though this is clearly not a preferred approach.

Then I found How do I allow an entity reference to be null in my custom entity? where one of the answer states that fields included in the entity_keys annotation must be set, regardless of the value set with setRequired().

Where is this kind of information documented?

This isn't documented in Creating a content entity type in Drupal 8. In fact, I had the impression it was mandatory to list all the fields in this annotation array.

Answer 1

The Drupal documentation - as the Drupal source code itself - has been provided by the contribution of the Drupal developer community. You will find, that it oftentimes isn't 100% complete nor up-to-date, as many of the developers ensure that code is working and more or less documented in the source code, but neglect the online documentation a bit while going on solving other issues or implementing different features. The documentation you mentioned even has a banner stating

This documentation needs work. See "Help improve this page" in the sidebar.

An experienced Drupal developer "just knows" such details from previous encounters with similar issues. If not sure, such a developer would also search the Drupal code base to see how the properties are used. Thanks to generally applied coding standards, extended source code documentation, and search features of modern IDEs, it's quite easy to read the Drupal code for any PHP developer; even easier for those that work with Drupal on a daily base.

If the source code doesn't offer answers, this StackExchange group or using a proper search engine may help to answer some Drupal questions relatively quick as well. The best explanation I found using a simple search is a citation from Daniel Siphos excellent "Drupal 8 Module Development" book:

Entity keys
The entity API defines a set of keys that are consistent across all entity types and by which common entity information can be retrieved. Since not all entity types need to have the same fields for storing that data, there is a mapping that can be done in the annotation for these

Which means, that the annotation property entity_keys is there to translate commonly used properties of entities (such as their ID, label or bundle) to their corresponding database fields that may be named differently for any entity type (e.g., nid/pid/..., title/name/..., type/variant/...). This allows the entity API to define common getter methods like ::id(), ::getLabel() or ::bundle() that can be used throughout the entire Drupal application in entity listing classes, the fields API, condition plugins, and so on, and therefore shouldn't be empty.

If you ever encounter a piece of Drupal documentation that appears to be incomplete, is missing cruicial information, or lacks information that the original contributors of the documentation page didn't deem worth mentioning, you can

• Log in, click Edit, and edit this page
• Log in, click Discuss, update the Page status value, and suggest an improvement
• Log in and create a Documentation issue with your suggestion

(Shamelessly copied from the sidebar of beforementioned article in the Drupal documentation. ;)

Answer 2

There isn't any documentation about that.
In SqlContentEntityStorageSchema::getSharedTableFieldSchema(), there is the following comment. (The relevant part is For now, we only hardcode 'not null' to a few "entity keys", in order to keep their indexes optimized.)

// A shared table contains rows for entities where the field is empty
// (since other fields stored in the same table might not be empty), thus
// the only columns that can be 'not null' are those for required
// properties of required fields. For now, we only hardcode 'not null' to a
// few "entity keys", in order to keep their indexes optimized.
// @todo Fix this in https://www.drupal.org/node/2841291. 

Fix NOT NULL handling in the entity storage and 'primary key' changes when updating the storage definition of an identifier field has been opened to remove the code that mark as NOT NULL some of the entity keys given for an entity type.

As a side note, the current workaround isn't to manually edit the database schema, but to override the getSharedTableFieldSchema() method implemented by the entity storage_schema handler.
This is what done by the File entity type, for example.

handlers = {
  "storage" = "Drupal\file\FileStorage",
  "storage_schema" = "Drupal\file\FileStorageSchema",
  "access" = "Drupal\file\FileAccessControlHandler",
  "views_data" = "Drupal\file\FileViewsData",
},
base_table = "file_managed",
entity_keys = {
  "id" = "fid",
  "label" = "filename",
  "langcode" = "langcode",
  "uuid" = "uuid",
  "owner" = "uid",
}

Drupal\file\FileStorageSchema::getSharedTableFieldSchema() contains the following code.

// Entity keys automatically have not null assigned to TRUE, but for the
// file entity, NULL is a valid value for uid.
if ($field_name === 'uid') {
  $schema['fields']['uid']['not null'] = FALSE;
}
return $schema;