I have a brand new site install profile I'm working on and with a clean site install, I'm seeing different results of the same permission setting. We have roles "Site Editor" and "Site Admin" and for panels and panelizer, all permissions are set identically. However, Site Admin is seeing all Settings, Context, Layout, Content, and Breadcrumbs. Site Editor is seeing what should be seen: Layout and Content. I'm not sure if there is some other hidden permission that a Site Admin has that the Site Editor does not have that is causing this override.
It appears that this is a documented bug in panelizer where a role having "Administer Organic groups permissions" can see all options for panelizer even when those panelizer permissions are not set for that role.