In theory, every routed path should be able to have an alias because it runs through \Drupal\Core\PathProcessor\PathProcessorAlias::processInbound() before routing, which converts it back to the system path.
However, private files, and image styles are special and quite weird. Because the routing system does not really support having an argument with / and doesn't know how to properly map that to a single argument (Not sure if that is still true or if there would be a better way now with more recent symfony versions), we have a workaround.
That workaround for private files is \Drupal\system\PathProcessor\PathProcessorFiles::processInbound(), which converts the full private path to just "/system/files" with file as a query argument.
The problem is now that those two run on the same event, and PathProcessorFiles runs first, at that point, the path is still the alias and does not match system/files/$something.
I'd say this is a bug and the priority of the service definitions should be adjusted for this. That is path_processor.files (currently 200) and path_processor_alias ( currently 100). Setting the first to 99 and clearing caches makes it work for me.
I'd suggest you open a core issue for this, I suspect nobody tried this yet.