You can create a View or Views with pages to accomplish this. The presentation of nodes connected to a taxonomy term on its display page is baked into Drupal separately from Views, I think. I'm not positive about that; I never use it. I'm more likely to go around it for reasons similar to what you're describing.
One page can easily present a list of of your location taxonomy terms. I would use a field-based view, and show the name of the term from the list. Uncheck the box that has the field link to the term directly, and instead check the box for the field to function as a custom link (under "Rewrite Results"), put in some URL that will include the location name, e.g., /user-location/{{ term_name }}
. Scroll further down and you'll see "Replacement Patterns". Use what seems appropriate there.
The other page may be field-based or entity-based dependent on your specific needs. It keys off users, and makes use of a contextual filter. That contextual filter is the location term name taken from the URL. The page URL will use the one you set in the first page. Thus this page will present user info filtered by the location term.