The normal behavior, described also in Give access to private files to certain role? seem to be the following.
Once configured, files stored in the private directory are inaccessible via a direct link; however, if Drupal constructs a link to the file, the file will be accessible to anyone who can see the link.
Does it work for media type fields? How can it be configured?
I would like the files to be accessible only from users who authenticate themselves using CAS.