Views plugins are testable, and Drupal core has a lot of examples. I believe most aren't UnitTestCase (or \PHPUnit_Framework_TestCase), but rather KernelTestBase tests.
KTB tests will boot a minimal Drupal setup so you don't have to set up extensive mocking for dependencies, especially config and database.
Views in particular has ViewsKernelTestBase, which does basic setup for testing Views plugins. You can use your IDE to find usages of this for example, but I know some are in web/core/modules/datetime/tests/src/Kernel/Views/
Essentially, you make a test module that has YAML for your test views. The test declares this in its
$modules list, along with the module that is providing the plugin. You also define the test views, and these get imported. You can then use the Views API in your test methods.
Then, as long as you use the new ViewsKernelTestBase, you can run it directly through PhpUnit, assuming you have your phpunit.xml setup properly.