Typical, no sooner than writing up my question do I find an answer. Though I'd love to hear if there are better ideas.
I discovered that you can use php to check the contexts in visibility settings. I didn't want to do this because I don't want to give the site admin access to php. However, I can make a couple of ctools access rulesets and make those available to the site admin to control visibility.
I haven't tried it yet but I think this will work, as long as the ctools context plugins work as I think they should.
[edit: it worked]