I have a private group with a "notebook"...most pages of that notebook may be accessed by all members of that group, but some pages shouldn't. I tried to control access with the module "Content access" but that only works with node types that are not part of any group because organic groups seems to grant access to all nodes if the user belongs to that group. Is there a way to deny access (based on roles) to nodes that belong to a group if the user is in the group?
Install the Module Grants module. It will solve the conflict. Actually, it will take the conflict into account, which Drupal does not.
Module Grants makes modules that deal with content access permissions operate better on unpublished (as well as published) content. It also makes sure that access grants behave in an orderly fashion when such modules are used together.
You can use the hook_node_access function for this. You can build an array of all your nodes and check this every time a user (linked with his role) access a node. You can allow, deny or ignore the request.
If you can't figure it out yourself, provide me with some info and I'll make you an example.