Vocabularies are bundles of the entity type taxonomy term. If you want to reference a vocabulary then you still need to set taxonomy_term as target type. Then define the vocabulary as target bundle:

$definition = BaseFieldDefinition::create('entity_reference')
  ->setSetting('target_type', 'taxonomy_term')
  ->setSetting('handler_settings', ['target_bundles' => ['custom_vocabulary' => 'custom_vocabulary']]);

See change record https://www.drupal.org/node/2576151

Vocabularies are bundles of the entity type taxonomy term. If you want to reference a vocabulary then first set taxonomy_term as target type and second restrict the field to a specific vocabulary as target bundle:

$definition = BaseFieldDefinition::create('entity_reference')
  ->setSetting('target_type', 'taxonomy_term')
  ->setSetting('handler_settings', ['target_bundles' => ['custom_vocabulary' => 'custom_vocabulary']]);

See change record https://www.drupal.org/node/2576151