If I have administrative access to a Drupal site, I can see the list of the modules enabled for the site.
How can I differentiate between third-party and custom modules?
You can just do this by looking at sites/all/modules folder - if the sitebuilder was a good one you will see two folders "custom" and "contrib". In modules administration page there is no simple way to differentiate between them.
Since custom modules are not hosted on Drupal.org, their .info files will not containing their version info. A module hosted on drupal.org (for example, the Entity API module) will contain the following information that a custom module will not have.
; Information added by Drupal.org packaging script on 2016-03-17
version = "7.x-1.7"
core = "7.x"
project = "entity"
datestamp = "1458222244"
So, in the Version column shown in admin/modules, you will not see any version number.
The User flag module you see in the screenshot is a custom module I created.
version = "7.x-1.7"
in the .info file?
Use Hacked module to differentiate.
This module scans the currently installed Drupal, contributed modules and themes, re-downloads them and determines if they have been changed. Changes are marked clearly and if the diff module is installed then Hacked! will allow you to see the exact lines that have changed.
If it is a custom module you can see that module in different color from "/admin/reports/hacked"
Or Use Module filter and have all of your custom modules in Custom package. Then you will be able to see all of the custom modules in Custom tab on admin/modules page.
If the Update manager module is enabled, you can simply go to the Available updates page (/admin/reports/updates).
Custom modules, not hosted by drupal.org, will not have information available and will be shown in grey.