OG provides no access control on attached files.  I ran into the same situation and came up with two options.

The [file depot module][1] integrates nicely with OG and does prevent users who are not members of a group from accessing files that are attached to a group.

> The filedepot module is full featured Document Management module that has a google docs like feel. It fulfills the need for an integrated file management module supporting role and user based security. Documents can be saved outside the Drupal public directory to protect documents for safe access and distribution.

Another option is writing a custom module that implements [`hook_file_download()`][2].  In that module you would have to get the node id of the file attachment then use that to determine what group the node belongs too then from there determine if the logged in user is a member of the group.  If they are not a member of the group the module would return -1 which would give them access denied if they try to directly download the file via url.  


  [1]: https://drupal.org/project/filedepot
  [2]: https://api.drupal.org/api/drupal/modules%21system%21system.api.php/function/hook_file_download/7