This issue - the results displayed by a view changing depending on the results of the filters - appears to be down to my having created two filters on the same taxonomy field - an unexposed field, and an exposed field.  I removed the unexposed filter. 

Instead of filtering twice on the same field, I created a new text field on the relevant content types.  I created a Rule to copy the name of my Category field in to the new field  Before Saving node.  

I hid the new text field with Field Permissions module.  Finally I created a Views Bulk Operations view to allow me to re-save all my existing nodes in one go, populating the new text field. 

I now have one field I can use in an unexposed filter to create my initial view, and a different but similar field that I can use to provide an exposed filter. 

This works, so I'm leaving it here as an answer, but it seems quite convoluted and messy, so I'm not accepting it as the ideal answer in the hope that someone may volunteer a better solution to the problem of needing to filter twice without running into weird filter order problems.