The Simple LDAP module's Simple LDAP Role submodule probably allows you to do something that would achieve the result you want (only a few people have more access than anonymous users). You can create a role in Drupal for privileged users, map that role to a group in LDAP, and then map everyone else to Authenticated user. If the authenticated user role has the same permissions as Anonymous you will have a very similar effect. That module might also allow you to control which groups can authenticate at all.
To have more features and more control you likely will need to create a custom submodule or switch to the larger LDAP module. The LDAP Module is powerful and useful, but not the best supported module in the community.