If a field is set a hidden on Manage display, unless it's displayed in a view, it cannot be seen anywhere on the site.
The format column contains the options to show the field value in different formats. Hidden is the global value which is present in all the format option list. If you choose hidden, it will hide the whole field from showing on the content page.
I understand it hides a field from the view mode, but you can also hide it if you don't try to do by using those modules, which are designed to output fields, like the Views module.
Is there still any possibility that users can see the hidden field even if those modules are not used?