Managed to get this working a different way, though Change G's way looks a lot simpler. If you want to go pure entity this way worked for me.
I created an entity validation plugin, based on this blog article (https://www.drupalwatchdog.com/volume-5/issue-2/introducing-drupal-8s-entity-validation-api). The comment module in core modules has a working example of this.
One issue I ran into is that the comment CommentNameContraintValidator class declares a $context class property; I can't figure out how this works for that class, but when creating my own it overrode the $context class property in the ConstraintValidator abstract class you need to extend from. When I removed the $context class property, it correctly used the parent $context class property and worked.
My guess is that it has to do with the ContainerInjectionInterface, which the CommentNameConstraintValidator implements and I didn't. I've stopped digging for now though.