All the examples for creating custom block online return #markup from build(), but I'd like to return a render array and have it use a template that's part of the module.
Would love an example of how to do this.
An array with #markup is a render array, just the most basic version of it. If you want to use a theme or template, then just do so. Using and defining templates hasn't changed that much compared to D7, the biggest difference is that they're now html.twig files and not .tpl.php.