Vocabularies are bundles of the entity type taxonomy term. If you want to reference a vocabulary then you still need to set `taxonomy_term` as target type. Then define the vocabulary as target bundle:

    $definition = BaseFieldDefinition::create('entity_reference')
      ->setSetting('target_type', 'taxonomy_term')
      ->setSetting('handler_settings', ['target_bundles' => ['custom_vocabulary' => 'custom_vocabulary']]);

See change record https://www.drupal.org/node/2576151